Module 6 ended on an awkward question: the platform is secure, but do we know whether it works? In a monolith, "it works" meant looking at one process and one log. Kilometre Zero has six services, three databases, a Kafka cluster, Redis, MinIO, a gateway and an IdP, spread across dozens of containers: nobody can watch all of that at once, and when something goes wrong it rarely shouts about it. An inventory service that answers in 900 ms instead of 40 is still "working"; a consumer of orders.events that has fallen three hours behind raises no exception; a Vault certificate that expires at 03:00 on a Sunday only gets noticed when the first order of the morning fails. This lesson is about making the platform observable through its first pillar, metrics: what to measure, how to collect it with Prometheus, how to put into numbers what Kilometre Zero promises Anna, and how to raise the alarm before she finds out. Logs and traces, the other two pillars, are the subject of lesson 07-02.

1. Observability and its three pillars
2. What to measure: infrastructure, platform, application and business
3. Metric types and the cardinality problem
4. Prometheus: the pull model, exporters and discovery
5. Essential PromQL: rates, percentiles and aggregations
6. SLI, SLO, SLA and the error budget
7. Alerts: symptoms, burn rate and Alertmanager
8. Grafana dashboards and active monitoring
9. Common mistakes and tips
10. Exercises and solutions
11. Conclusion

Monitoring means checking things we already know can fail: the disk fills up, the process dies, latency climbs. Observability is a property of the system: the ability to answer questions we had not anticipated ("why have Lleida orders containing fresh-cheese been taking twice as long since yesterday?") from what the system emits to the outside. In a distributed system the difference matters because unanticipated questions are the norm: the interesting failures are combinations of parts, not isolated parts.

The three signals a system emits, which together provide observability, are:

Metrics are the starting point because they are the only one of the three that scales with the number of series, not with traffic: counting 10 orders per minute costs the same as counting 10,000. They are the signal that fires alerts and the one that draws the Grape Harvest Week dashboard. Logs and traces come in afterwards, once the metric says "something is wrong" and you need to find out what.

A common mistake is to start with infrastructure (CPU, memory) because that is what agents give you out of the box, and end up with a hundred graphs that don't tell you whether Anna can buy anything. It is better to think in four layers, from the bottom up, and pick a few meaningful metrics in each.

This is what node_exporter measures on each machine or cAdvisor on each container: CPU, memory, disk (space and I/O latency), network (bytes, errors, dropped packets). Brendan Gregg's USE method applies to these resources: for each one, Utilisation (percentage of time busy), Saturation (queued work that doesn't fit: run queue, swap, disk queue) and Errors.

The pieces in the middle have metrics of their own that give early warning of application problems:

• Kafka: the lag of each consumer group (messages produced minus messages consumed, per partition). Growing lag in analytics is tolerable; in delivery it means the dashboard shows stale positions for van-3. Also partitions without an in-sync replica (covered in 07-03).
• PostgreSQL: active connections against the maximum (max_connections), transactions per second, how far the orders-db-replica replica lags behind the primary, table sizes and locks being waited on.
• Redis: the hit ratio of the catalogue cache (hits / (hits + misses)), memory used against the limit, evicted keys, command latency.
• Cassandra: read and write latency per table, pending compactions, nodes seen as down by gossip.

In Site Reliability Engineering, Google popularised the four golden signals, the ones to watch on any service that handles requests:

1. Latency: how long it takes to respond, distinguishing successful requests from failed ones (a fast error is not "good latency").
2. Traffic: how much demand it receives (requests per second, messages consumed per second).
3. Errors: what fraction of requests fail, whether explicitly (5xx, gRPC UNAVAILABLE) or implicitly (it answered 200 but took longer than agreed).
4. Saturation: how "full" the service is: busy threads, pool connections in use, work queue.

The RED method (Tom Wilkie) is the version for services: Rate, Errors, Duration. It matches the first three golden signals and is what we instrument uniformly across every service in km0/.

Golden signals per service:

Business metrics are the ones that tell Jordan Hall and Martha Hill whether the platform is serving its purpose, and they often catch failures that technical metrics miss: if orders answers 200 in 80 ms but orders per minute drop to zero at 11:00 on a Saturday, something upstream has broken (the front end, Kong, Keycloak). For Kilometre Zero: orders created per minute (per market and per producer), payment rejection rate, average order value, failed stock reservations per product, on-time deliveries. They are instrumented exactly like the technical ones, in the same Prometheus.

Prometheus (and OpenTelemetry Metrics, which uses the same conceptual model) distinguishes four types:

Two clarifications that prevent a lot of mistakes:

• A counter is never read raw. km0_orders_total is 1,284,522 and that tells you nothing; what matters is its derivative: rate(km0_orders_total[5m]) = orders per second over the last 5 minutes. Prometheus handles restarts (when the counter goes down, it assumes a restart and does not produce a negative rate).
• Histogram versus summary: the histogram stores how many observations fell below each bound (le="0.1", le="0.25", le="0.5", ...). Percentiles are computed on the server and, above all, can be aggregated across instances: the p99 of orders with 3 replicas is obtained by adding up buckets. A summary computes the p99 in each process, and there is no correct way to combine three p99s (the average of percentiles is not a percentile). In a distributed system, barring very specific cases, you use a histogram.

Each distinct combination of name + labels is a time series that Prometheus keeps in memory and on disk. km0_request_duration_seconds{service, method, code} with 6 services × 10 methods × 5 codes × 12 buckets = 3,600 series: fine. If someone decides to add order_id as a label, every order creates a new series per bucket: a million orders means twelve million series, and Prometheus dies. The rule: a label may only take a small, bounded set of values (service, method, status code, market, order status). Never entity identifiers (order_id, user, request_id), client IP addresses, or paths with parameters that haven't been normalised (/orders/P-2026-000123 must be recorded as /orders/{id}). Anything that needs the identifier goes to logs or traces (07-02).

An interesting borderline case is km0_stock_units{product}: with a few hundred products it is acceptable; with a hundred thousand it is not. At Kilometre Zero it is limited to the products in active campaigns and, for the rest, a gauge aggregated per producer is published.

Prometheus works in pull mode: it doesn't wait for services to send it data; instead, every 15 seconds (configurable) it issues a GET /metrics to each target and stores what it finds. The format is plain text:

Pulling has important consequences in a distributed system:

• Prometheus knows when a target isn't responding: the metric up{job="orders", instance="orders-2:8000"} is 0. With a push model, a dead service simply stops sending and you have to infer the absence.
• Services don't need to know Prometheus's address or manage send queues.
• It needs to discover its targets: in docker-compose listing them is enough; in Kubernetes (07-05) it discovers them by asking the API; in the cloud, by asking the provider's API. For short-lived jobs (a Job that lasts 3 seconds) there is the Pushgateway, which is the exception, not the rule.

Components that don't expose /metrics themselves are covered by exporters: small processes that translate the state of PostgreSQL, Kafka, Redis and so on into the Prometheus format. At Kilometre Zero:

Every service shares one module with the common metrics, so that names and labels are identical and the dashboards work for any of them.

Points worth understanding:

• Counter, Gauge and Histogram are process-wide global registrations: they are declared once, at module level, and any part of the service uses them. prometheus_client exposes them all together on /metrics.
• .labels(...) selects the specific series; the first time a combination is used, it is created. That is why you have to control which values get passed in (cardinality).
• measure records the code even when there is an exception: without that except, requests that fail with an internal error would not show up in the histogram, and latency would look better than it is.
• REQUESTS_IN_FLIGHT is the cheapest saturation signal to obtain: how many requests there are at once. If in orders it exceeds the number of server threads, the next ones wait in a queue.

inventory already has AuthInterceptor and ServiceInterceptor (06-04). We add one more, which wraps every call and turns the gRPC status code into a label:

And at server start-up the order of the interceptors matters: metrics first, so that calls rejected by authentication are measured too.

orders is a FastAPI application; a middleware measures every request and /metrics is mounted on the same app:

Using the route template (/orders/{id}) instead of the real URL is the cardinality rule put into practice.

In docker-compose.yml the observability stack is added alongside the services:

The PostgreSQL exporter user is read-only on the pg_stat_* views; in a real deployment its credentials would come from Vault (06-04).

PromQL is the query language. Four constructs cover 90% of uses:

rate(counter[window]): per-second rate over the window. Requests per second to orders, summing the three replicas and broken down by method:

The histogram's _count suffix (number of observations) is used as a request counter: every measured request is one observation.

Error rate: the fraction of requests with a 5xx code:

histogram_quantile(q, buckets): a percentile computed from the buckets. The p99 of orders, aggregating every instance:

The sum by (le) is mandatory: the buckets of all instances must be added up while preserving the le label (the bucket bound), which is what histogram_quantile needs in order to interpolate.

increase(counter[window]): how much it has gone up over the window (useful for "orders in the last hour"):

Others that come up daily: avg_over_time(gauge[10m]) to smooth a gauge, absent(up{job="orders"}) to detect that there are no targets at all, topk(5, ...) for the five products with the least stock, and operators between vectors (/, and, unless) that match series by their labels.

With 1,000 requests at 50 ms and 10 at 5 s, the average is 99 ms: "all good". The p99 is 5 s: one customer in a hundred waits five seconds. In a distributed system, where one catalogue page makes several internal calls, tail latency is amplified: if each call has a 1% chance of being slow and a request makes 10 of them, 10% of requests see at least one slow call. That is why targets are set on percentiles (p50 for "the normal case", p99 for "what somebody suffers every minute") and never on averages.

In 01-03 we worked out the "nines": a service with 99.9% availability can be down for 43 minutes a month. That calculation talked about "being down"; now we have the instruments to define it precisely.

• SLI (Service Level Indicator): a concrete measurement of the service, expressed as the fraction of good events over the total. "The fraction of requests to POST /orders that respond without a 5xx in under 500 ms."
• SLO (Service Level Objective): the internal target for that SLI over a window. "99.9% of requests, measured over 30-day windows."
• SLA (Service Level Agreement): the contractual commitment, with consequences (penalties). Always looser than the SLO: Kilometre Zero promises producers 99.5% and holds itself to 99.9%.

The SLOs of the orders service:

Notice how the histogram lets you express the latency SLO as a fraction: "how many observations fell into the le="0.5" bucket" divided by the total is exactly "the fraction of requests under 500 ms". This only works if 0.5 is a bucket bound, which is why LATENCY_BUCKETS includes 0.5 and not, say, 0.4 and 0.6.

A 99.9% SLO means that 0.1% of bad requests is allowed: that is the error budget. With 2 million requests a month, that is 2,000 errors. The budget turns reliability into a currency you can spend:

• If there is budget left, the team deploys quickly, tries things out, runs chaos experiments (07-06).
• If the budget runs out (a 40-minute incident uses up nearly all of it), non-urgent deployments are frozen and the effort goes into reliability.

It is also what gives alerts their meaning: a spike of errors doesn't matter in itself; what matters is how fast the budget is being consumed.

An alert should wake someone up only when a user is suffering or is about to, and that someone must be able to do something about it. "CPU at 85% on inventory-2" meets neither condition: it may be normal during Grape Harvest Week, and it doesn't say what to do. "2% of orders have been failing for 5 minutes" does. Causes (CPU, memory, lag) go on dashboards, for diagnosis when the symptom alert fires, or become low-priority alerts (a ticket, not a phone call).

The reasonable exceptions are predictive alerts on causes with a known deadline: a certificate that expires in 12 h or a disk that will fill up in 4 h at the current rate (predict_linear) are warnings that prevent the symptom.

The burn rate is the speed at which the error budget is being consumed, relative to the "fair" speed. A burn rate of 1 means that, at the current pace, the 30-day budget runs out in exactly 30 days. A burn rate of 14.4 means it runs out in 2 days. The technique recommended by the SRE Workbook is to alert on two windows at once, a long one and a short one:

The long window stops a 30-second spike from waking anyone up; the short window makes the alert resolve as soon as the problem stops (without it, the 1 h alert would stay active for an hour after the fix).

About these rules:

• Recording rules (record:) compute the expensive expression once and store it as a new metric; alerts and dashboards reuse it. With 6 h windows over histograms, doing it this way saves Prometheus a great deal of work.
• for: requires the condition to hold for that long before firing. It is the first filter against alerts caused by flapping.
• Vault's 24 h certificates (06-04) make the 12 h alert the natural threshold: if a certificate hasn't been renewed halfway through its life, automatic renewal is broken.
• km0_vault_lease_seconds_remaining and km0_audit_failed_verifications_total are metrics that the services themselves publish: business and security instrumentation uses the same mechanism as latency instrumentation.

Prometheus evaluates and fires; Alertmanager decides whom to notify, how to group and when to keep quiet:

Silences: when Jordan is about to restart inventory-2 for a migration, he creates a 30-minute silence with a matcher (instance="inventory-2:9464") from the UI or with amtool silence add instance=inventory-2:9464 --duration=30m --comment="migration". Without silences, the alternative is ignoring alerts, which is where alert fatigue begins: when half the notifications are noise, nobody reads the one that matters. Rules of thumb: every page alert must have a runbook (07-03) and a possible action; an alert that fires and "fixes itself" three times in a row is downgraded to a ticket or removed; the number of pages per week is reviewed like any other metric.

A dashboard is not a collection of everything that can be graphed; it is a top-to-bottom narrative: first whether the business is doing well, then whether the services are meeting their targets, then why. For the Roble Alto Winery campaign:

Grafana is provisioned from files so that the dashboard lives in the repository, not in clicks:

And a JSON dashboard cut down to two panels, to show the structure:

Everything so far is passive: it measures whatever traffic there is. If there is no traffic at 04:00, the error rate is 0/0 and tells you nothing; if Kong has a broken route that nobody has used yet, it says nothing either. Active monitoring (synthetic probes) generates artificial traffic to check the whole path: the blackbox_exporter requests GET /api/v1/catalog/markets through Kong every 15 s and publishes probe_success, probe_duration_seconds and probe_ssl_earliest_cert_expiry. A more ambitious probe (a small script that creates and cancels a test order as the user [email protected], flagged so that it doesn't count in analytics) verifies the entire saga. These probes are the ones that catch Sunday's expired certificate before Anna does.

Everything we have seen uses prometheus_client directly. OpenTelemetry is the open standard that unifies metrics, logs and traces in one SDK and one protocol (OTLP): with it, km0_request_duration_seconds would be a Histogram from OTel's MeterProvider, exported to a Collector that hands it over to Prometheus (or to any other backend). The semantics (types, labels, cardinality, PromQL) are the same. Kilometre Zero keeps prometheus_client for metrics because it is simpler, and adopts OpenTelemetry for traces and log correlation in 07-02, which is where it adds the most value.

• Measuring infrastructure only. CPU and memory don't tell you whether Anna can buy. Start with RED on every service and two business metrics; add USE for resources to support diagnosis.
• High-cardinality labels. order_id, user, IPs or non-normalised paths multiply the series until Prometheus falls over. If you need it for an investigation, it goes to logs or traces.
• Summaries on replicated services. Quantiles computed on the client cannot be aggregated across instances. Use histograms with buckets that include the thresholds of your SLOs.
• Alerting on averages or on causes. The average hides the tail; CPU at 80% is not a problem in itself. Alert on user-facing SLIs and burn rate; everything else goes to dashboards or tickets.
• Alerts with no for: and no runbook. Every blip wakes someone up and nobody knows what to do. Every page alert has a for, an owner and a runbook linked in the annotation.
• Forgetting exceptions in the middleware. If the metric is only recorded when the request finishes successfully, the latency and errors of the ones that fail disappear. Record in finally.
• Relying on passive monitoring alone. No traffic, no signal. One synthetic probe per critical path (catalogue, create order, login) covers the quiet hours and untested configuration changes.
• Dashboards built by hand in Grafana. They get lost, they get duplicated, nobody knows which one is the good one. Provision them from the repository, next to the code they instrument.

Exercise 1. During Artisan Cheese Week, Montblanc Dairy asks how many orders containing aged-cheese were confirmed in the last hour, and Martha wants to know whether inventory is responding worse than yesterday. (a) Can the first question be answered with km0_orders_total{status="confirmed"} as it is currently defined? If not, what would you change and what cardinality risk would it carry? (b) Write the PromQL query that gives the p95 of ReserveStock over the last 5 minutes, aggregating the two inventory instances, and explain why a per-instance avg(histogram_quantile(...)) won't do. (c) Which gauge would tell you whether the problem lies in the inv-bcn/inv-vlc replicas rather than in the service?

Exercise 2. The orders SLO is 99.9% success over 30 days and the service receives an average of 3 million requests a month. (a) How many errors fit in the budget? (b) At 10:00 on Saturday an incident begins in which 8% of requests fail. With traffic at 100 requests/s, how long does it take to use up the entire budget? (c) Which of the two burn rate alerts (14.4 over 1 h/5 min; 6 over 6 h/30 min) will fire first, and roughly when, taking the for: clauses into account? (d) At 10:20 the incident is resolved; why does the alert resolve shortly afterwards even though the 1 h window is still contaminated?

Exercise 3. One Monday Jordan finds 47 notifications from the weekend in #km0-operations: 30 of PostgresHighConnections on the orders replica (which climbs to 90% every night during the km0_daily_sales DAG and comes back down by itself), 12 of TargetDown for delivery during a scheduled 10-minute deployment, and 5 of KafkaLagAnalytics. None of them corresponded to a real problem. For each group, propose the specific change (in alerts.yml, in alertmanager.yml or in the procedure) that avoids the noise without losing detection of a real problem, and explain which symptom signal would cover the case where it really was one.

Exercise 1.

(a) No: km0_orders_total only has the status label; it doesn't know which products each order contained. Two options: a separate counter km0_order_lines_total{product} incremented for every confirmed line (cardinality = number of products, acceptable if it is limited to campaign products or if the catalogue has a few hundred; with tens of thousands of products you would have to aggregate per producer or answer that question from analytics, not from Prometheus), or adding product to km0_orders_total, which is worse because an order with three products would be counted three times and would break the orders metric. Never order_id. (b) histogram_quantile(0.95, sum by (le) (rate(km0_request_duration_seconds_bucket{service="inventory", method="ReserveStock"}[5m]))). Adding up the buckets of both instances by le first rebuilds the joint distribution, and the percentile is computed over that; averaging two p95s computed separately does not give the p95 of the whole (if one instance handles 90% of the traffic with p95 = 40 ms and the other 10% with p95 = 800 ms, the average, 420 ms, describes nobody). (c) The replication lag published by postgres_exporter (pg_replication_lag_seconds or the equivalent over pg_stat_replication) for inv-bcn and inv-vlc; if that gauge goes up, reads on the replica return stale stock and inventory may be retrying or waiting; if it is at zero, the problem lies in the service or in the primary.

Exercise 2.

(a) 0.1% of 3,000,000 = 3,000 errors a month. (b) 100 requests/s × 8% = 8 errors/s; 3,000 / 8 = 375 s ≈ six and a quarter minutes. The whole month's budget is gone in a little over six minutes. (c) Burn rate = 0.08 / 0.001 = 80, far above both thresholds. The 5 min window exceeds 1.44% almost immediately; the 1 h window needs to accumulate: with the previous hour clean, the 1 h error rate reaches 1.44% about 11 minutes into the incident (0.08 × t / 60 min = 0.0144 → t ≈ 10.8 min); with for: 2m, OrdersFastBurnRate notifies at around 10:13. The 6 h one would need 0.006 × 360 / 0.08 = 27 minutes plus for: 15m, so it never fires if the incident lasts 20 minutes. (d) Because the rule requires both windows at once (and): once the incident is resolved, the 5 min window drops below the threshold within a few minutes, and even though the 1 h window stays above it until 11:20, the conjunction no longer holds and Alertmanager marks the alert as resolved. That is precisely the point of the short window.

Exercise 3.

PostgresHighConnections on the replica during the DAG: it is a cause, not a symptom, and it is expected behaviour; options: raise the threshold for the replica only with an unless/matcher on instance, add for: 30m (the DAG takes less than that), or better, downgrade it to severity: ticket (it already is) and route tickets to a daily digest instead of to Slack in real time (long group_interval/repeat_interval on the default route); the real symptom that should notify is the p99 latency of orders or the replication lag, which are covered by OrdersLatencyP99. TargetDown during deployments: the deployment procedure must create a silence (amtool silence add job=delivery --duration=15m --comment="deployment") as a pipeline step; in addition, with several replicas the alert should be "every instance of the job is down" (absent(up{job="delivery"} == 1) or count(up{job="delivery"} == 1) == 0), not a single instance; the symptom that covers a genuinely broken deployment is the lag on delivery.positions (KafkaLagDelivery) and the dashboard's synthetic probe. KafkaLagAnalytics: 200,000 with for: 30m is too sensitive for a batch consumer that catches up every night; raise the threshold, lengthen for to several hours, or change it to an alert on the analytics symptom: "the km0_daily_sales DAG hasn't finished by 07:00" (an Airflow metric) or "the last run failed". In every case the principle is the same: a page alert protects a specific user experience; everything else is information for diagnosis.

A distributed system doesn't tell you when it fails; you have to build the instruments. Observability rests on three signals and this lesson has developed the first: metrics, the only one that scales with the number of series and not with traffic. We have seen what to measure in four layers (infrastructure with USE, platform, application with RED and the golden signals, business), the four metric types and why the histogram is the right type for latencies on replicated services, the cardinality rule that forbids order_id as a label, Prometheus's pull model with its exporters and probes, and the indispensable PromQL (rate, histogram_quantile with sum by (le), increase). On that foundation, the "nines" from 01-03 have become concrete SLIs and SLOs for orders (99.9% success, p99 < 500 ms), with an error budget that gets spent and multi-window burn rate alerts that notify on symptoms, routed by Alertmanager to whoever can act, and a provisioned dashboard that tells the story of Grape Harvest Week from top to bottom.

Now, when OrdersFastBurnRate fires at 10:13 on a Saturday, Jordan will know that 8% of orders are failing. What the metrics won't tell him is which ones, or why: whether it is inventory rejecting reservations of crianza-wine, the payment gateway, or a traceparent that got lost in a Kafka header. For that you need the other two pillars: structured, centralised logs, which tell you what happened to order P-2026-000125, and distributed traces, which show in which service the 480 ms went. That is the next lesson.