The 06-03 evaluation answers "can I deploy this?" with cases you chose. This lesson answers the next question: what is happening right now in production, with the questions nobody chose? An LLM system fails in ways classic software doesn't know: it throws no exception when it hallucinates, returns no 500 when it answers half-heartedly, and can double its cost without anything "breaking". In 05-02 we planted traces.py::log_step() announcing that it would germinate here; time to deliver: define what to record on each request, structure the logs so they're exploitable, distill metrics and dashboards that tell the system's health at a glance, set up alerts that say what to do (not just that something's happening), and close the loop with user feedback feeding the evaluation set. It's the difference between operating DocuBot and praying for DocuBot.

1. From agent traces to system observability
2. What to record on each request
3. Structured logs: JSON per line with logging
4. What NOT to log
5. Aggregate metrics and dashboards
6. Actionable alerts: symptom, probable cause, action
7. User feedback: the continuous improvement loop
8. The tooling ecosystem, by category

log_step() was born in 05-02 with a modest mission: note down each iteration of the agent loop (what it thought, which tool it called, what it got) to enable debugging. That's a trace: the biography of one request. Full observability adds two more floors:

• Logs: individual structured events, queryable ("show me all of yesterday's requests with confidence < 0.3").
• Metrics: numeric aggregates over time ("fallback rate per hour", "accumulated cost for the day").

The practical distinction: traces explain one request; metrics watch over all of them; logs are the raw material for both. In an LLM application there's an added peculiarity: a good share of the "failures" are silent degradations (worse answers, higher costs, more "I don't know") that only show in the aggregates. That's why a system that only has debug traces is blind in production.

The design starts with the central event: one line per complete request (in addition to the per-agent-step lines that traces.py already emits). Fields, grouped by what they let you answer:

Notice the economy of the design: almost all of this already existed. estimate_cost() and measure_call() come from 03-04, the contract's confidence from 02-03, the similarities from retrieve() from 04-04, the tools and confirmations from 05-02, the pseudonymization from 06-02. Observability doesn't invent data; it collects, with discipline, the data the system already produces.

A free-text log ("User asked about billing, it went fine") can't be queried or aggregated. The de facto standard is one line = one JSON object (JSON Lines): a human reads it with grep, any log platform ingests it, and a twenty-line script aggregates it.

Implementation with the standard library's logging module:

And its integration at the exit point of docubot.py::answer (abridged version):

A resulting line (formatted here for readability; in the file it's a single line):

Teaching details in the code: the custom Formatter is the single place where the format is decided (changing it touches nothing else); the fields travel via extra to avoid string concatenation; ensure_ascii=False keeps non-ASCII characters readable; and request_id is the key that joins this line with the per-step traces from log_step() (which now includes it too) and with the user feedback that will arrive minutes later.

Logs are one more form of data processing (we saw it in 06-02: they fall within the scope of the GDPR and of rights requests). Rules for DocuBot:

• Neither the question nor the answer in raw form in the production log. They contain what the user typed — potentially unredacted personal data. Intermediate options, from less to more exposure: metadata only (shown above); text redacted with redact() (06-02) and short retention; full text only in development environments with fictional data. If the business needs to review real conversations, let that be an explicit decision with legal, with restricted access and an expiry — not a logger default.
• Real identities: always the session/user pseudonym, with the mapping table protected separately.
• API keys and secrets: never. Beware of cheerful dumps of configuration objects or exceptions that drag headers along.
• Full system prompts: prompts.py is versioned in git; prompt_version in the log is enough. Logging the entire prompt on every request is noise, cost and exposure of intellectual property.
• And the general rule: log references and measurements, not contents. {"id": "billing_doc_03#2", "sim": 0.71} instead of the chunk's text: the id lets you fetch the content if an investigation requires it, without duplicating it across the logging system.

The aggregates are computed on top of the logs. These are the indicators with the best signal-to-effort ratio for DocuBot, and what each one tells:

A reasonable dashboard for DocuBot is these 8 series with a comparison against the previous week. The golden rule when designing it: every chart must have an owner for the question "if this changes, what do I do?" — a dashboard nobody knows how to interpret is decoration.

An alert that only says "something is wrong" produces fatigue and ends up muted. An actionable alert carries the differential diagnosis built in — exactly the format of the 04-05 diagnostic table, now extended to the whole system:

Two principles when configuring thresholds: alert on trends and ratios (a rate that doubles) rather than on noisy absolute values; and every fired alert must end either in an action or in a threshold adjustment — alerts ignored three times get removed or fixed.

All the previous signals are indirect. The direct one is asking: a 👍/👎 at the foot of each answer (optionally with a comment on the 👎). Cheap to implement — an endpoint that attaches the verdict to the request_id — and extremely valuable if exploited well:

The naive use is computing "% thumbs up" and putting it on the dashboard (useful, with biases: a minority votes, and the unhappy vote more). The powerful use is closing the loop with 06-03:

1. Each 👎 is joined, via request_id, with its complete log: question (redacted), chunks, similarities, versions, answer.
2. Periodic review of the 👎: bad retrieval? Unfaithful answer? Documentation gap? The symptom table above, applied case by case.
3. Representative real failures become new cases in the 06-03 evaluation set, with their expected result annotated.
4. The next prompt/chunking iteration is validated against them already: that failure mode doesn't come back without the runner catching it.

This loop is the mature answer to a question hovering over the course since 02-04: "where do good evaluation cases come from?". At first, from your imagination and collected questions; in steady state, from real production failures. Observability isn't just surveillance: it's the quarry for the evaluation set.

Everything above is built with the standard library on purpose: understand the mechanism before adopting the platform (the same philosophy as the hand-rolled agent loop of 05-03). In a growing system, evaluate by category:

• LLM-specific observability platforms (commercial and open source): they automatically capture call/agent traces with prompts, tokens and costs; they offer conversation visualization, integrated evaluations and dataset annotation. They deliver a lot of 06-03 + 06-04 pre-packaged; in exchange, your prompts and data pass through yet another platform — the 06-02 analysis applies to them too.
• General-purpose observability + OpenTelemetry: the cloud world's open standard for traces/metrics/logs. Semantic conventions exist for generative AI, so LLM calls appear as spans in the same system where you already observe the rest of your platform. Attractive if your organization already runs this stack: DocuBot stops being an island.
• Classic log aggregators: if you already have one, your JSON Lines go in as-is; this lesson's dashboards and alerts get built on top with nothing AI-specific.

An honest selection criterion: the differential value of the LLM platforms lies in agent trace visualization and in integrating evaluation and annotation; if your structured logs already answer your questions, don't rush to add infrastructure. The important part — what to record, what not to, what to look at and what to do about it — is what you've designed in this lesson, and it's portable to any tool.

• Logging everything "just in case". Full prompts and raw conversations are cost, noise and a privacy problem (06-02). References and measurements, not contents; and defined retention.
• Classic software observability without LLM signals. Uptime and HTTP errors don't see hallucinations, confidence degradation or documentation gaps. The specific metrics (anti-hallucination phrase, confidence, rejected confirmations) are what detect the silent failures.
• Logs without versions. If every line doesn't carry prompt_version/chunking_version/model, you won't be able to answer the only question that matters after a deploy: "did this start with the change?".
• Watching the mean latency. The mean hides the tail; the p95 user is the one who leaves. Percentiles always (you were already computing them with measure_call() in 03-04).
• Alerts with no associated action. Every alert must be born with its symptom→cause→action table row. If nobody knows what to do when it fires, it's not an alert: it's scheduled noise.
• Collecting feedback and not exploiting it. The thumbs percentage on a dashboard is the small part; the value lies in turning the 👎 into cases for the 06-03 set. Feedback without a loop is a bottomless suggestion box.
• Confusing monitoring with evaluation. The dashboards tell you something got worse; the 06-03 runner tells you whether your fix corrects it without breaking something else. They're the two halves of the same muscle: production feeds the set; the set protects production.

It's Monday. DocuBot's dashboard shows, versus last week: no_info_phrase rate stable (11%), mean confidence stable, daily cost +40%, latency p95 +35%, TTFT stable, exact cache rate down from 30% to 8%, request volume +5%. No new version was deployed on Friday. What's your main hypothesis, which log fields would confirm it, and what would you look at first?

Add the human-in-the-loop information to log_request: a confirmation field worth null if the request proposed no actions, "accepted" or "rejected" if it did. Also write the conceptual query (in pseudocode or Python over the JSON lines) that computes the weekly rejection rate, and explain which alert in the table it connects to.

A user votes 👎 on an answer in the permissions category with the comment "it told me guests can edit tasks, and that's not true". The log shows: confidence: 0.82, two retrieved chunks with similarities 0.58 and 0.47, no_info_phrase: false. Describe the diagnosis step by step (what you'd look at and in what order) and write the resulting evaluation case for the 06-03 set.

Exercise 1:

The key piece is the exact cache drop (30% → 8%) with almost flat volume and no deploy: every request that used to come free out of ResponseCache now goes to the model — that simultaneously explains the cost (+40%) and the p95 (+35%: cached answers were instantaneous and lowered the percentile), with TTFT stable (the provider is fine; there are simply more real calls). Main hypothesis: the question pattern changed — new questions that aren't in the cache. Why on a Monday? A typical candidate: Nubelia launched something over the weekend and unprecedented questions are arriving. Confirmation in the logs: distribution of cache (misses spiking), volume per category (did one category grow?), and sampling the new questions (redacted). Fine-grained note: a stable no_info_phrase suggests the wiki does cover the new topics — if it were rising too, the undocumented-launch hypothesis would gain strength. First action: look at the cache distribution per category; if it's a legitimate new topic, the extra cost is real demand (and the cache will warm itself back up within days).

Exercise 2:

In answer, when logging:

Query over the JSON Lines:

Important detail: the denominator is the requests with a proposal (confirmation is not None), not all of them — that's why the field distinguishes null from the other two values. It connects with the alert "rejected-confirmation rate 5% → 30%": a high rejection rate = the agent proposes badly (degraded prompt/tool descriptions) or, in the worst case, hostile content pushing actions — which points back to the per-step traces and the 06-01 defenses.

Exercise 3:

Ordered diagnosis: (1) With the request_id, retrieve the full trace and the chunk ids (0.58 and 0.47 — the second grazes RELEVANCE_THRESHOLD=0.45: mediocre retrieval). (2) Read the chunks by their id: does any of them claim guests can edit tasks? Two branches: if no chunk says so → unfaithful answer (a hallucination with confidence 0.82: the dangerous profile), a generation problem; if an outdated chunk says so → the problem is content/ingestion, not the model — fix the document and re-ingest (04-03). (3) Depending on the branch: unfaithfulness branch → a case for the runner with the faithfulness judge; document branch → editorial fix + verification. Evaluation case (unfaithfulness branch):

With this, the next prompt or chunking variation is also validated against the real failure: the continuous improvement loop, closed.

The traces planted in 05-02 have germinated into a complete nervous system: every request leaves a JSON line with its pseudonymized identity (the 06-02 lesson applied to the logs themselves), its prompt/chunking/model versions (without them no post-deploy analysis is possible), its chunks and similarities, its tools and confirmations, its tokens, cost and latency percentiles, and the quality signals the system was already emitting for free — the contract's confidence and the anti-hallucination phrase, whose rate turns out to be the best thermometer of the documentation's gaps. On top of the logs, eight metrics with an owner and an alert table that doesn't scream "something is wrong" but says "this is what's probably happening and this is the first thing to look at". And the closing of the loop: real users' thumbs down become, via request_id, new cases in the 06-03 set — production feeds the evaluation, the evaluation protects production. With this, the helmet is complete: DocuBot resists manipulation (06-01), respects data (06-02), is evaluated before changing (06-03) and is watched while it operates (06-04). Only one thing remains, and it's the most satisfying: bringing together all the pieces built across six modules, looking at the complete architecture at once, walking a real request end to end and going through the final production checklist. The next lesson is the final project — DocuBot, whole.