The Project | About Us | Contribute | Donations | License

HOME

This lesson settles a debt announced twice: in 02-04, when we built the set of 30-50 real questions and the binary rubric with a promise to automate its execution, and in 04-05, when the rubric grew to 9 criteria and we said "06-03 automates this". DocuBot is now protected (06-01) and respectful with data (06-02), but it's still being evaluated by hand: someone fires off the questions, reads the answers and fills in a CSV. That doesn't scale, doesn't run before every deployment, and depends on the evaluator's mood. Here we assemble the full testing pyramid for LLM applications: classic pytest for your deterministic code, contract tests over the model's outputs, a runner that executes the entire set and records versioned results, and an LLM acting as a judge for the criteria no assert can verify — calibrated against your manual evaluation, which becomes the gold standard.

The problem: why classic testing isn't enough (and why it's still necessary)
The three testing levels in an LLM application
Level 1: deterministic unit tests with pytest
Level 2: contract and property tests over LLM outputs
Level 3: the quality evaluation runner
LLM-as-judge: automating the non-deterministic criteria
Regressions: evaluate before deploying
The cost of evaluating

The Problem: Why Classic Testing Isn't Enough (and Why It's Still Necessary)

Traditional testing rests on a premise: same input → same output. LLMs break it (non-determinism was limitation number 4 in module 1): the same question can produce different answers, all acceptable — or not. An assert answer == expected over free text will fail on Tuesday and pass on Thursday.

The wrong reaction is to give up ("it can't be tested, we'll just try things by hand"). The right one is to realize that an LLM application has two territories:

Your code: clean_json, validate_contract, the chunking, run_tool, redact... 100% deterministic. It's tested with plain old pytest, without spending a single token.
The model's outputs: non-deterministic in their literal form, but with properties that must always hold (the JSON is valid, the sources exist, the anti-hallucination phrase appears when it should) and with qualities that require judgment (is it faithful to the documentation? is it relevant?).

Each territory has its tool. Confusing them produces the two classic mistakes: spending tokens testing a parser (expensive, slow, unnecessary) or trying to verify faithfulness with an equality assert (impossible).

The Three Testing Levels in an LLM Application

Level	What it verifies	Deterministic	Tool	When it runs	Cost
1. Unit tests of your own code	Parsers, validators, chunking, redaction, tools with mocks	Yes	Plain pytest	On every commit (seconds)	Zero tokens
2. Contract/property tests	Invariant properties of LLM outputs: valid JSON, contract honored, anti-hallucination phrase when applicable, never invents sources	The verification is; the output isn't	pytest + real model calls	Before deploying, with a few cases	Low
3. Quality evaluation	Faithfulness, relevance, correctness over the complete case set	No	Runner + automatic criteria + LLM-as-judge	Before every prompt/model/chunking change; periodically	The one to manage

The pyramid shape is no accident: many cheap tests at the bottom, few expensive ones at the top. Most of DocuBot's bugs throughout the course (JSON with surrounding text, badly cut chunks, invalid arguments) live at level 1, where finding them costs milliseconds.

Level 1: Deterministic Unit Tests with pytest

Nothing new under the sun — and that's the good news: half of DocuBot is tested like any Python project. Representative examples:

# test_validation.py
import pytest
from validation import clean_json, validate_contract

def test_clean_json_extracts_from_noisy_text():
    raw = 'Sure, here you go:\n{"category": "api", "answer": "...", ' \
          '"sources": [], "confidence": 0.4}\nHope that helps.'
    assert clean_json(raw)["category"] == "api"

def test_validate_contract_rejects_unknown_category():
    output = {"category": "martian", "answer": "x",
              "sources": [], "confidence": 0.9}
    assert not validate_contract(output)

def test_validate_contract_rejects_confidence_out_of_range():
    output = {"category": "api", "answer": "x",
              "sources": [], "confidence": 1.7}
    assert not validate_contract(output)

# test_chunking.py — chunking is a pure function: text in, chunks out
from chunking import chunk_markdown

def test_no_chunk_exceeds_the_maximum():
    text = "## Heading\n" + ("filler paragraph. " * 500)
    assert all(len(c["text"]) <= 1800 for c in chunk_markdown(text))

def test_overlap_preserves_context():
    # two consecutive chunks share the ~300 characters of overlap
    chunks = chunk_markdown("## A\n" + "x" * 4000)
    assert chunks[0]["text"][-100:] in chunks[1]["text"]

And the key pattern for the module 5 tools — mocks: you test your plumbing without calling the LLM or the real API.

# test_tools.py
from tools import run_tool

def test_nonexistent_project_returns_is_error(monkeypatch):
    # nubelia_api is already a mock, but we pin its state for the test
    execution = run_tool("query_project",
                         {"name": "nonexistent"})
    assert execution["is_error"] is True   # the 05-01 pattern

def test_argument_validation_stops_before_executing():
    # the 06-01 allowlist, tested in milliseconds
    execution = run_tool("create_support_ticket",
                         {"priority": "apocalyptic",
                          "title": "t", "description": "d"})
    assert execution["is_error"] is True

Notice the dividend of the course's architecture decisions: since the agent loop is yours (05-03) and the logic lives in separate modules, each piece is tested in isolation. The 06-01 defense (validate_arguments) and the 06-02 redaction (redact) are pure functions: they enter this level with tests that are trivial to write.

Level 2: Contract and Property Tests over LLM Outputs

Here we do call the model, but we don't ask "is this the exact answer?" — we ask "does it satisfy the properties every answer must satisfy?". It's the idea of property-based testing adapted: you don't know the output, but you know its invariants.

DocuBot's three canonical properties:

# test_llm_contract.py — these run with real calls: few and carefully chosen
import json
import pytest
from rag import answer_with_rag, NO_INFO_PHRASE
from validation import clean_json, validate_contract
from vectordb import get_existing_sources  # real ids from docs_nubelia

# Property 1: the output always honors the JSON contract
@pytest.mark.llm  # marker so they can be excluded from the fast cycle: pytest -m "not llm"
def test_output_honors_contract():
    output = clean_json(answer_with_rag("How do I change my billing plan?"))
    assert validate_contract(output)

# Property 2: without relevant documentation, the anti-hallucination phrase appears
@pytest.mark.llm
def test_unanswerable_question_triggers_no_info_phrase():
    output = clean_json(answer_with_rag(
        "What is the remote-work policy on Mars?"))  # doesn't exist in the wiki
    assert NO_INFO_PHRASE in output["answer"]
    assert output["confidence"] <= 0.3

# Property 3: never invents sources — every cited source exists in the collection
@pytest.mark.llm
def test_cited_sources_exist():
    output = clean_json(answer_with_rag("How do I create a webhook?"))
    existing = get_existing_sources()
    assert all(s in existing for s in output["sources"])

Two engineering nuances:

Non-determinism doesn't go away: a property may hold in 2 out of 3 runs. That's why the 04-05 habit (3 runs per case) also applies here for flaky tests: either run N times and require N/N (hard properties like the contract), or accept a threshold — and then the case belongs to level 3, not this one.
Few cases, carefully chosen: this level doesn't traverse the entire set (that's level 3); it uses a handful of sentinel cases — one per property, perhaps one per classifier category — so it can run often without budget pain.

Level 3: The Quality Evaluation Runner

We reach the promised automation. The runner is a script that: (1) loads the complete set — the 30-50 cases from 02-04 with their 4 types (easy ~40%, ambiguous ~25%, no answer ~20%, soft adversarial ~15%), the annotated chunks from 04-05 and the security adversarials from 06-01 —; (2) runs each case against the system; (3) validates what's automatable; (4) delegates to the judge what isn't; (5) records a CSV with all the versions.

# eval_runner.py
import csv, json, time, uuid
from datetime import date

from rag_dataset import RAG_DATASET                       # cases with annotated chunks (04-05)
from adversarial_cases import SECURITY_ADVERSARIAL_CASES  # 06-01
from rag import answer_with_rag, NO_INFO_PHRASE
from agent import agent_loop
from tools import TOOLS
from validation import clean_json, validate_contract
from prompts import PROMPT_VERSION
from chunking import CHUNKING_VERSION
from client import MODEL
from pricing import estimate_cost

RUNS_PER_CASE = 3   # the 04-05 habit: non-determinism gets sampled

def run_case(case: dict) -> dict:
    """Runs a case through its corresponding path and evaluates automatic criteria."""
    start = time.time()
    if case.get("path") == "agent":              # multi-step or tool cases
        raw = agent_loop(case["question"], TOOLS, max_iterations=8)
    else:                                        # default path: RAG pipeline
        raw = answer_with_rag(case["question"])
    duration = time.time() - start

    output = clean_json(raw)
    results = {
        "contract_valid": validate_contract(output) if output else False,
        "duration_s": round(duration, 2),
    }
    if output:
        # Automatic criteria: deterministic, no judge
        if case["type"] == "no_answer":
            results["no_info_phrase_ok"] = NO_INFO_PHRASE in output["answer"]
        if "expected_sources" in case:           # thanks to the annotated chunks
            cited = set(output["sources"])
            results["sources_ok"] = cited <= set(case["expected_sources"])
        results["output"] = output
    return results

def run_dataset():
    run_id = f"{date.today()}-{uuid.uuid4().hex[:8]}"
    cases = RAG_DATASET + SECURITY_ADVERSARIAL_CASES
    rows = []
    for case in cases:
        for n in range(RUNS_PER_CASE):
            r = run_case(case)
            rows.append({
                "run_id": run_id,
                "case_id": case["id"], "type": case["type"], "run": n + 1,
                "prompt_version": PROMPT_VERSION,
                "chunking_version": CHUNKING_VERSION,   # "headings-v1-1800-300"
                "model": MODEL,
                **{k: v for k, v in r.items() if k != "output"},
            })
    with open(f"eval_{run_id}.csv", "w", newline="", encoding="utf-8") as f:
        w = csv.DictWriter(f, fieldnames=rows[0].keys())
        w.writeheader(); w.writerows(rows)
    return rows

Points worth internalizing:

The versions go on every row. PROMPT_VERSION, CHUNKING_VERSION and MODEL turn the CSV into a reproducible experiment: a month from now you'll know exactly which combination produced which numbers. It's the same principle as the 02-04 CSV, now automated.
Each case travels through its real path (answer_with_rag or agent_loop), because you evaluate the system you deploy, not a laboratory version.
Automatic criteria first. Valid contract, anti-hallucination phrase in the no_answer cases, sources within the expected set (possible thanks to the 04-05 chunk annotation): all of that is settled with cheap asserts. The judge only steps in where the assert can't reach.
The retrieval metrics (recall@k, precision@k, MRR) are not recomputed here: they live in eval_rag.py (04-05) and evaluate the retriever in isolation. This runner evaluates the system end to end; they're complementary diagnostics (if the runner degrades, eval_rag.py tells you whether the culprit is retrieval or generation).

LLM-as-Judge: Automating the Non-Deterministic Criteria

What remains are the criteria of the 04-05 nine-point rubric that no assert can check: is the answer faithful to the chunks (adding nothing that isn't there)? Is it relevant to the question? The solution with the best cost/utility ratio is using another LLM as an evaluator: LLM-as-judge.

The idea: the judge receives the question, the retrieved chunks and the answer, and applies the rubric issuing binary verdicts (yes/no) — the same scale you used by hand, because 1-to-10 scales produce inconsistent judges and pseudo-precise numbers.

# judge.py
import json
from client import client, JUDGE_MODEL, extract_text   # it can be a small model
from retries import call_with_retries

JUDGE_PROMPT = """You are a strict evaluator of a documentation assistant.
Evaluate the ANSWER using only the provided DOCUMENTATION as reference.

Criteria (answer exactly true or false for each one):
- faithfulness: every claim in the answer is supported by the documentation.
  If the answer adds facts, steps or figures that don't appear, it is false.
- relevance: the answer directly addresses the user's question.

Return ONLY a JSON: {"faithfulness": bool, "relevance": bool,
"justification": "one sentence per criterion"}

QUESTION:
{question}

DOCUMENTATION (sole source of truth):
{documentation}

ANSWER TO EVALUATE:
{answer}"""

def judge(question: str, documentation: str, answer: str) -> dict:
    prompt = JUDGE_PROMPT.format(question=question,
                                 documentation=documentation,
                                 answer=answer)
    output = call_with_retries(client, JUDGE_MODEL,
                               [{"role": "user", "content": prompt}])
    return json.loads(extract_text(output))

The detail of the prompt requesting a justification isn't cosmetic: it forces the judge to "reason" before the verdict (the CoT of 02-02 applied to evaluation) and gives you material for auditing its failures.

Calibration: Don't Trust the Judge Without Measuring It

A judge is just another LLM: it makes mistakes too. Before delegating anything to it, you calibrate it against the gold standard — your manual evaluations from 02-04 and 04-05, which stop being historical baggage and become the asset that validates the judge:

Take 30-50 cases you already evaluated by hand with the rubric.
Run them through the judge.
Measure the % agreement per criterion (and, if you want rigor, a kappa-style coefficient that discounts agreement by chance).
Rule of thumb: with agreement ≥ 85-90% on a criterion, the judge can automate it (spot-checking the disagreements by sampling); below that, iterate the judge's prompt or keep that criterion manual.

Known Judge Biases and Mitigations

Bias	What it does	Mitigation
Position	In A/B comparisons it systematically prefers the first (or last) option presented	Evaluate each answer in isolation against the rubric (as our judge does); if you compare pairs, evaluate in both orders and discard if they don't match
Length	Favors longer, more detailed answers even when they're not better	Binary criteria anchored to the documentation ("supported by", not "complete"); include in the calibration cases where the short answer is the correct one
Self-preference	Scores outputs generated by its own model family higher	Use as judge a model different from the one evaluated when possible; calibration against the human gold standard detects the bias even if you can't avoid it
Leniency	Leans toward "yes" when in doubt	Explicit strictness instruction ("if it adds facts that don't appear, it is false") and few-shot examples of negative verdicts in the judge's prompt

Regressions: Evaluate Before Deploying

Everything above crystallizes into one discipline: no prompt, model or chunking change is deployed without passing the evaluation. It's the LLM equivalent of "you don't merge with red tests", and the reason we've been versioning everything since 02-04.

The flow, at a conceptual level (a gate in CI):

flowchart LR
    A[Change: PROMPT_VERSION,<br>CHUNKING_VERSION or MODEL] --> B[Level 1: pytest<br>seconds, zero tokens]
    B -->|green| C[Level 2: contract tests<br>sentinel cases]
    C -->|green| D[Level 3: runner + judge<br>full set]
    D --> E{Meets thresholds and<br>doesn't degrade the baseline?}
    E -->|yes| F[Deploy and archive CSV<br>as the new baseline]
    E -->|no| G[Review disagreements<br>and iterate the change]

The acceptance thresholds are defined in advance (not after seeing the numbers, which is how regressions get legalized). A reasonable example for DocuBot:

Valid JSON contract: 100% (it's code + a hard property; no excuses).
Anti-hallucination phrase in no_answer cases: ≥ 95%.
06-01 adversarial cases passed: 100% of the deterministic ones (the allowlist doesn't negotiate), ≥ 90% of the judged ones.
Faithfulness (judge): ≥ 90% and never more than 2 points below the previous baseline.

That second kind of threshold — comparing against the previous run, not just against an absolute minimum — is what detects regressions: the archived CSV of the last good version is your baseline, and the version columns tell you exactly what to compare against what.

The Cost of Evaluating

A complete run has a price you already know how to estimate with estimate_cost() (03-04): ~45 cases × 3 runs × (1 call to the system + 1 to the judge) ≈ 270 calls. Levers to make evaluating painless:

A small model as judge. Verifying against a rubric is easier than generating; small models perform well as judges of binary criteria and cost a fraction. Calibration against the gold standard tells you whether the small one suffices — it's a measured decision, not a hunch.
Staging: levels 1 and 2 always run (free or nearly); the full level 3 runs on prompt/model/chunking changes and periodically.
Sampling: for the fast iteration cycle, a stratified subset (keeping the 40/25/20/15 proportions of the case types) gives cheap directional signal; the full set is reserved for the deployment decision.
Judge only where needed: the automatic criteria filter first; if the contract is already invalid, no judge is spent evaluating faithfulness.

Order of magnitude with the prices from pricing.py: a full run with a small judge typically lands in cents to a few euros — compared with the ~€417/month of operation projected in 03-04, the anti-regression insurance is one of the system's cheapest line items.

Common Mistakes and Tips

Testing the LLM with exact-equality asserts. assert answer == "..." over free text is a guaranteed flaky test. Over the model's outputs you verify properties (contract, phrase, sources), not literals.
Spending tokens testing your own code. clean_json doesn't need an LLM to be tested; it needs 10 twisted cases in pytest. Everything at its level of the pyramid.
Trusting the judge without calibrating it. A judge with no % agreement measured against the gold standard is another source of pretty, useless numbers. First agreement ≥ 85-90%, then trust — and even then, periodic sampling of its verdicts.
Evaluating with 1 run per case. Non-determinism turns a single run into a coin flip. Three runs (the 04-05 habit) distinguish "always fails" from "sometimes fails", which are different bugs.
Forgetting the versions in the record. A CSV without PROMPT_VERSION/CHUNKING_VERSION/MODEL is an orphan number: you can't compare, you can't reproduce, you can't bisect regressions.
Defining thresholds after seeing the results. That's the fast lane for normalizing regressions. Threshold first, run afterwards — and the previous baseline as a mandatory reference.
Confusing offline evaluation with monitoring. Everything in this lesson happens before deploying, with known cases. What happens in production with real questions from real users is the territory of 06-04.

Exercises

Exercise 1: Placing Verifications in the Pyramid

For each verification, decide which level it belongs to (1, 2 or 3) and with which mechanism it's checked (deterministic assert, property with a real call, automatic runner criterion or LLM-as-judge): (a) estimate_cost() correctly computes the price of 1,000 input tokens and 500 output tokens; (b) every DocuBot answer is parseable JSON that passes validate_contract; (c) answers to billing questions are faithful to the retrieved documentation; (d) redact() replaces two different emails in the same text; (e) given an adversarial case with an intruding instruction in the chunk, the answer doesn't repeat the false claim.

Exercise 2: Writing a New Property Test

Write a level 2 test for this DocuBot property: "if the answer has confidence ≥ 0.7, then sources cannot be empty" (a confident answer without sources smells like a hallucination). Use answer_with_rag, clean_json and a question that does have an answer in the wiki.

Exercise 3: Designing the Judge's Calibration

You have 40 cases evaluated by hand for the faithfulness criterion (gold standard: 31 "yes", 9 "no"). You run the judge and get: it matches on 29 of the "yes" and on 5 of the "no". (a) Compute the overall % agreement. (b) What proportion of the human "no" does the judge detect, and why does that number matter more than the overall agreement for this criterion? (c) Would you automate this criterion already? What would you do first?

Solutions

Exercise 1:

(a) Level 1, deterministic assert: estimate_cost is pure arithmetic; pytest without tokens.
(b) Level 2, property with real calls over sentinel cases (and also an automatic criterion on every row of the level 3 runner — hard properties get checked in both places because at level 3 they come for free).
(c) Level 3, LLM-as-judge: faithfulness requires semantically comparing answer and documentation; no assert can do that.
(d) Level 1, deterministic assert: redact is a pure function from 06-02.
(e) Level 3, runner criterion: partly automatable by searching for the false claim in the answer (deterministic) and completed with the judge for the nuances — exactly as the success_criterion was designed in 06-01.

Exercise 2:

import pytest
from rag import answer_with_rag
from validation import clean_json

@pytest.mark.llm
def test_high_confidence_implies_sources():
    """Property: confidence >= 0.7 => non-empty sources.
    A 'confident' answer without sources is the typical profile of a hallucination."""
    output = clean_json(answer_with_rag("How do I create a webhook in Nubelia?"))
    assert output is not None, "the output is not parseable JSON"
    if output["confidence"] >= 0.7:
        assert output["sources"], (
            f"confidence {output['confidence']} without sources: possible hallucination"
        )

Notice the logical form: the property is an implication ("if high confidence, then sources"), so the test doesn't fail if the model returns low confidence — that would be a different case, not a violation. It's a common pattern in property tests: verify the invariant only when its precondition holds (and have cases that guarantee the precondition holds often).

Exercise 3:

(a) Agreements: 29 + 5 = 34 out of 40 → 85% overall agreement.
(b) Of the 9 human "no" (unfaithful answers), the judge detects 5 → 55.6%. It matters more than the overall 85% because the "no" cases are precisely what the judge exists to catch: an unfaithful answer that passes as good is the expensive error (a hallucination in production), while the overall agreement is inflated by the majority of easy "yes" cases. It's the classic imbalanced-classes problem: look at the minority class's recall, not just accuracy.
(c) Not yet. The 85% grazes the threshold, but detecting only half of the unfaithful answers is insufficient. First: read the 4 "no" the judge let slip (the judge's justifications help), reinforce the judge's prompt with strictness instructions and 1-2 few-shot examples of negative verdicts similar to the missed ones, perhaps try another model as judge, and recalibrate. Automate only when the detection of "no" is consistently high — and even then, keep sampling disagreements.

Conclusion

The debt from 02-04 and 04-05 is settled: DocuBot's evaluation no longer depends on a heroic afternoon with a CSV, but on a pyramid that runs on its own — pytest for the deterministic code (parsers, chunking, argument validation, redaction: half the system gets tested for free), property tests for the invariants of the model's outputs (JSON contract, anti-hallucination phrase, sources that exist), and the runner that executes the complete set — easy, ambiguous, no-answer, soft adversarials and the 06-01 security adversarials — recording every row with PROMPT_VERSION, CHUNKING_VERSION and MODEL so each number has a full name. Where the assert can't reach, an LLM-as-judge with the 04-05 rubric on a binary scale issues the faithfulness and relevance verdicts — but only after earning trust against your manual gold standard, and watched for its known biases. With thresholds defined in advance and the previous baseline as the measuring stick, "can I deploy this prompt change?" goes from an act of faith to a traffic light. But this traffic light only sees the questions you imagined: on deployment day begin the questions nobody imagined, the real costs, the Monday-afternoon latencies and the users who leave without saying why. Seeing what happens in there — turning the traces we planted in 05-02 into real observability, with structured logs, metrics, dashboards and actionable alerts — is the course's penultimate lesson.

Evaluating and Testing LLM Applications

Contents