Throughout the module we have been postponing one decision: what "going well" means. We have metrics (06-01), traces (06-02), resilience patterns (06-03) and autoscaling (06-04), but nobody has yet said how many errors on POST /v1/orders are acceptable, how long the saga may take before someone must look, or whose phone rings when the DLQ fills up at three in the morning. Without those answers, observability is a collection of pretty dashboards consulted after the disaster. This lesson turns it into an operating system: SLOs that translate the customer's experience into numbers, alerts that fire only when those numbers are at risk, on-call rotations and runbooks that know what to do, and a way to manage incidents and learn from them without looking for someone to blame.

1. SLI, SLO and SLA: definitions and error budget
2. Choosing TechCorp's SLIs
3. Concrete SLOs and their monthly budget
4. Expressing the SLIs in PromQL with recording rules
5. SLO-based alerts: multi-window burn rate
6. Symptom-based alerts, and why not to alert on causes
7. Alertmanager: routes per team, grouping, silences and inhibition
8. On-call and runbooks
9. Incident management: roles, severities and communication
10. Blameless postmortem: INC-2031
11. SLO review and error budget policy
12. Summary table of TechCorp's alerts

Rules: the SLA is always looser than the SLO (if the SLO is 99.9%, the SLA 99.5%: the margin is what avoids paying penalties); an SLI is expressed as a proportion (0-1) so it can be summed, compared and turned into budgets; and a 100% SLO does not exist: each additional nine multiplies the cost and no dependency (the cloud, the PSP) offers it.

The error budget is the part of the SLO that is allowed to fail: 1 − SLO. With a 99.9% SLO over 30 days, 0.1% of requests may fail; expressed in time, if the service were completely down, 43 minutes per month. It is the most useful idea in the lesson: it turns "reliability" into a currency that is spent on deployments, experiments and accidents, and that, when exhausted, forces you to stop (section 11).

A good SLI measures what the customer perceives, not what is convenient for engineering. Ana Ruiz does not know what CPU is; she knows whether her order was created, whether the products page loads fast and whether the confirmation reaches her. TechCorp's four SLIs, one per question:

Design notes:

• The availability SLI is measured at the gateway (what the customer sees), not at Orders: if Orders is healthy but the gateway does not route, the customer suffers all the same. 4xx do not count as failures: a 422 for invalid data is the client's fault; a 503 because Catalog is down (06-03) does count, even if it is "someone else's".
• The latency one is defined as a proportion of requests under a threshold, not as "the p95 < 300 ms". The proportion is a clean SLI (good/total) and directly exploits the histogram's le="0.3" bucket; the p95 is still watched in Grafana.
• The saga one needs the histogram from 06-01 and excludes from the denominator the orders cancelled for OUT_OF_STOCK or PAYMENT_REJECTED (resolving quickly with a rejection is also "on time": saga_duration_seconds is observed when the order closes in any final status).
• The outbox one is an internal SLI: the customer does not see it, but when it fails the saga one will fail minutes later. A good SLO system has few customer-facing SLIs and some internal ones that warn earlier.

Why those numbers and not others: 99.9% on orders because every minute without selling costs measurable money (Black Friday, 01-05) and because the infrastructure from 05-02/05-04 (2 replicas, rolling without downtime) makes it achievable; 99.5% on latency and saga because 0.5% of slow pages or late confirmations is annoying but does not prevent buying, and because they depend on the PSP and on MongoDB at peak. The minutes math: 30 days × 24 h × 60 min × (1 − SLO); for 99.9%: 43,200 × 0.001 = 43.2 min. And a warning: minutes are an intuition; the real budget is measured in requests. A 100% outage for 43 minutes at 4 a.m. consumes far less budget (few requests) than 5% errors for 8 hours during business hours.

Computing histogram_quantile or rate divisions over 30 days on every query is expensive. Recording rules precompute each SLI into a new series, with a conventional name sli:<name>:ratio_rate<window>, and the alerts and panels query that series:

• Each record rule stores the result of expr as a new series every interval. The alerts in the next section combine the 5 m, 30 m, 1 h and 6 h windows; the "remaining budget" panel uses the 30 d one.
• The failure proportions (errors, slow, late) are recorded rather than the success ones: burn rates are computed over failure.
• The latency one uses 1 − (good/total): bucket{le="0.3"} counts those that took ≤ 300 ms.
• slo:orders_budget_remaining:ratio is the metric shown to the business: "we have 62% of this month's budget left".

The 30-day windows require enough retention in Prometheus (or Thanos/Mimir for the long term, a mention). The four SLIs are also documented as text in techcorp/platform/observability/slos.md: definition, owner, target and review date.

The naive alert "error rate > 1% for 5 minutes" has two problems: it fires on a two-minute spike that barely spends any budget, and it does not fire on a sustained 0.5% over three days that does exhaust it. The SRE book's alternative is to alert on the speed at which the budget is consumed (burn rate): 1× means that at the current pace the budget runs out exactly at the end of the 30-day window; 14.4× means it would run out in 2 days (30/14.4). Two windows are combined (a long one to make sure it is real, a short one so the alert clears soon after it is resolved):

For the 99.9% SLO (budget 0.001), 14.4× equals an error rate of 1.44% and 6× equals 0.6%:

• expr with and: both windows must exceed the threshold at the same time. The 1 h one keeps a spike of seconds from waking anyone; the 5 m one makes the alert resolve minutes after the cause is fixed (without it, it would stay active until the whole hour cleared).
• for: 2m: the condition must hold for two minutes before firing; against the noise of a single scrape.
• labels: severity decides the channel (section 7); team decides the recipient; slo groups.
• annotations: what the on-call person reads at three in the morning. Always runbook and dashboard; an alert without a runbook is not deployed (Platform rule, checked by a linter in CI).

The same two alerts are replicated for catalog latency (CatalogLatencyBurnFast, budget 0.005: thresholds 7.2% and 3% slow) and the saga (SagaLateBurnFast, over sli:saga_late). With four SLOs and two alerts each, that is eight SLO alerts in total: few, meaningful and all tied to something the customer notices.

SLO alerts cover "the customer is suffering". There is a second group of symptom-based alerts that anticipate that they will suffer or that something is broken even though the customer does not notice yet; they must be few and have a clear threshold:

(The templated team in DlqHasMessages is a simplification; in practice there is one rule per queue with its owning team from 02-01.) Others of the same kind: CertificateExpiring (probe_ssl_earliest_cert_expiry from Blackbox Exporter, < 14 days, a mention), ReadyProbeFailing for more than 10 minutes, RabbitMQNodeDown.

What is not paged on: causes. "CPU > 80%", "memory > 90%", "disk at 70%" may mean a problem or an HPA doing its job (06-04); if the customer is not suffering and no symptom appears, there is nothing to do at three in the morning. They are left as panels and, at most, as severity: info to a Slack channel that is read in the morning. The discipline is this: page → the customer is suffering or will within minutes; ticket → something is wrong and must be fixed this week; info → good to know. Every alert that wakes someone up and requires no action is an alert to delete or downgrade.

Prometheus evaluates the rules; Alertmanager decides whom to notify and how. The Platform team's configuration (alertmanager-config secret, versioned with the chart values):

• Routing by severity and team: critical goes to PagerDuty (on-call) and to the team's Slack (continue: true); warning only to the team's Slack. The four channels correspond to the four teams from 02-01.
• Grouping (group_by): if 12 Catalog pods enter CrashLoopBackOff, one notification arrives with 12 alerts, not 12 messages.
• Silences: during planned maintenance (a RabbitMQ migration on a Sunday) a silence is created in the Alertmanager UI with matchers (team="orders", 2 h) and a comment; alerts are evaluated but not sent.
• Inhibition: a "root" alert (RabbitMQNodeDown) suppresses the derived ones (OutboxStuck, DlqHasMessages) so that on-call receives one notice, not ten.
• Slack and PagerDuty are fictional in the course; the pattern works just the same with Opsgenie, Teams or email.

TechCorp, with ~25 engineers, sets up one weekly on-call rotation among the developers of the four teams (not just Platform: whoever builds, operates), with a primary and a secondary person, an agreed compensation schedule and one rule: on-call only receives critical alerts with a runbook. Everything else waits for business hours.

A runbook is the page opened from the alert that says what to look at and what to do, written for someone who does not know the service and is half asleep. Template and example for OrdersErrorBudgetBurnFast:

Every alert in the table in section 12 has its own in techcorp/platform/runbooks/, versioned in Git and with a last-reviewed date: a runbook with commands that no longer work is worse than none. The golden rule is that whoever resolves an incident without a runbook writes it when done.

An incident is any situation that degrades the service and requires coordination. When a page fires, the on-call person does three things: acknowledges the alert (so it does not keep escalating), opens a channel (#inc-2031-payments-stock in Slack) and assesses the severity:

The roles in a SEV1/SEV2, so that not everyone types at once:

• Incident commander: coordinates, decides and assigns; does not debug personally. Usually the on-call person until someone with more context explicitly takes over ("I'm taking command").
• Communicator: keeps the business and support informed (a message every 30 min in #platform-status with: what is happening, impact, what is being done, next update) so that nobody interrupts those resolving it.
• Operators/investigators: one or several; they follow the runbook, execute changes and note every action with its time in the channel.

Principles: stabilize before understanding (roll back the deployment, scale replicas, disable the canary, reprocess later: the root cause is sought once the customer is no longer suffering); a real-time timeline in the channel (the postmortem will reconstruct it); and communication with the business in their language ("orders cannot be created since 10:14; we estimate recovery in 20 min; orders already created are not affected"), never in ours ("payments.stock is stuck because of a poison message").

Every SEV1 and SEV2 ends with a postmortem in the following week. "Blameless" means it is assumed that people acted reasonably with the information they had, and that the question is what in the system (code, tools, processes, alerts) allowed it to happen and to last as long as it did. If the postmortem looks for someone to scold, next time the information gets hidden. Template and summarized example:

The actions become tasks in the backlog with an owner and a date, and are reviewed at the next operations meeting; a postmortem whose actions are not carried out is a useless document. And it is shared with the whole company: incidents are the most expensive source of learning there is; wasting it is a double loss.

SLOs are not forever. Every quarter, each team reviews with product: does the SLI measure what the customer notices? Is the target too ambitious (always red, nobody believes it) or too loose (always 100%, it says nothing)? Have expectations changed (a new marketplace demands more)? Targets are adjusted, useless SLIs retired and missing ones added.

And the error budget is used with a written policy, agreed between Marta and the teams:

It is the balance that was missing in the velocity debate of 05-03: the DORA metrics push toward deploying more often; the error budget says up to when. A team with budget to spare should dare more (more deployments, more experiments), not less: not spending the budget is also a waste.

Eleven alerts (plus the per-queue variants). If in three months one of them has never required action, it is downgraded or removed; if an incident was not detected by any, the one that would have caught it is added.

• A hundred alerts. Each one seems reasonable; together they cause fatigue and on-call ignores them all. Start with the SLO ones and four symptoms; add only from postmortems.
• Alerting on a fixed error threshold ("> 1% for 5 m"). Noise on spikes, silence on slow degradations. Multi-window burn rate.
• A 99.99% SLO "because it sounds good". That is 4 minutes a month: incompatible with dependency rolling updates, with the PSP and with the company's budget. Choose what the business needs and the architecture allows.
• Measuring the SLI where it is convenient and not where the customer is. Orders at 100% with the gateway down is a met SLO and a closed store.
• Alerts without a runbook. Waking someone up to start from scratch. Runbook or it is not deployed.
• Postmortems with names and adjectives ("X deployed without testing"). Next time nobody will tell what happened. System, not people.
• Postmortem actions without owner or date. Forgotten within a week. Tasks in the backlog, reviewed.
• On-call only in Platform. Developers do not see the consequences of their code. Whoever builds, operates (with training and compensation).
• Tip: build a single "SLO status" dashboard with the four proportions and the remaining budget, and put it on the team's screen; it is the conversation with product that was missing.
• Tip: test the alerts: amtool alert add OrdersErrorBudgetBurnFast severity=critical team=orders sends a test alert and checks the routing without waiting for a real incident.

Define an SLI and an SLO for notifications-service ("the customer receives the confirmation email"), state which metric it would come from (you may propose a new one consistent with 06-01), compute the monthly budget and justify a severity for its alert.

Write the complete SagaLateBurnFast rule (expression with the necessary recording rules, for, labels, annotations) for the 99.5% SLO, and explain why its threshold is 7.2% and not 1.44%.

It is 02:10. OutboxStuck fires (outbox_pending = 340) and, at the same time, DlqHasMessages on inventory.orders.dlq with 2 messages. POST /v1/orders responds 202 as usual. Classify the severity, decide which alert you handle first and describe the first five minutes in the style of the runbook.

Exercise 1

SLI: proportion of confirmation emails handed to the provider in under 5 minutes from order.confirmed, over the total of order.confirmed received. Metric: histogram notification_latency_seconds{type="confirmation"} observed in Notifications from the event's occurred_at until the provider's 2xx response, with buckets [1, 5, 30, 60, 300, 900]; SLI = rate(..._bucket{le="300"}) / rate(..._count). SLO: 99% over 30 days (a late email is annoying, it does not prevent buying; the external provider does not guarantee more). Budget: 1% ≈ 432 minutes of "total outage" per month, or about 900 late confirmations out of 90,000. Alert: NotificationsLateBurnFast with severity: warning (not critical): it wakes nobody up; emails are retried from the queue (06-03) and arrive when the provider comes back. If the delay exceeded hours it would become SEV3/SEV2 for the impact on support, but not a page.

Exercise 2

Recording rules needed: sli:saga_late:ratio_rate1h (already defined) and sli:saga_late:ratio_rate5m (same expression with [5m]).

The threshold is burn rate × budget: 14.4 × 0.005 = 0.072 (7.2%). With a 99.5% SLO the budget is five times larger than with 99.9%, and therefore the failure rate that exhausts it "in two days" is also five times larger. The burn rate is the same (14.4×); the absolute percentage depends on the SLO.

Exercise 3

Severity: POST /v1/orders keeps accepting orders, but none of them progresses (events do not leave the outbox): the saga is stopped for everyone buying right now, and in 10 minutes the watchdog will not act (the orders are in PENDING, not in STOCK_RESERVED), so unconfirmed orders will pile up. It is a SEV2 (broken flow with growing impact; not SEV1 because there is no data loss: the outbox keeps everything). OutboxStuck is handled first: it is critical, it is the most "upstream" cause, and Inventory's DLQ with 2 messages is warning and can wait (it might also be a consequence: if RabbitMQ is misbehaving, both alerts fit). Five minutes: (1) acknowledge in PagerDuty, open #inc-XXXX-outbox; (2) Grafana "Orders saga": outbox_pending climbing in a straight line since 01:55, orders_created_total normal, rabbitmq_queue_messages_ready{queue="inventory.orders"} = 0 → the relay is not publishing; (3) kubectl -n techcorp get pods -l app=orders-service → 2 pods Running; Loki {app="orders-service"} | json | message=~"relay.*|rabbit.*" → "channel closed by the broker" at 01:55 and no "batch published" since then: the relay lost its channel and did not reconnect (a reconnection bug); (4) immediate remedy: kubectl -n techcorp rollout restart deployment/orders-service (the new pods open a channel, the relay drains the outbox in a minute; the rolling update from 05-04 does not interrupt POST); check that outbox_pending drops; (5) note the timeline, look at the 2 messages in inventory.orders.dlq (probably related: events published halfway before the cut; they are reprocessed with reprocessDlq.js once the relay is running), message to the communicator/#platform-status and, in the morning, a postmortem with the action "the relay must reconnect with backoff and /health/ready must fail if the channel is closed" (06-03).

With this lesson TechCorp stops "looking at dashboards" and starts operating against objectives. Four SLIs say what the customer notices (availability of POST /v1/orders, latency of GET /v1/products under 300 ms, sagas confirmed in under 60 s, outbox freshness), with 99.9%/99.5% SLOs and their error budget in minutes and in requests; the sli:*:ratio_rate<window> recording rules precompute them; the multi-window burn rate alerts (OrdersErrorBudgetBurnFast at 14.4× over 1 h/5 m, …Slow at 6× over 6 h/30 m) and a handful of symptom alerts (OutboxStuck, DlqHasMessages, CircuitOpen, PodInCrashLoop) reach the owning team through Alertmanager and, only if critical, on-call; each with a runbook; incidents are managed with SEV1-SEV3 severities, a commander and a communicator, and end in a blameless postmortem whose actions are tasks; and the error budget policy decides when to deploy and when to stop. This closes the monitoring and maintenance module: TechCorp's services emit structured logs and RED and business metrics (06-01), traces that join the HTTP request with the saga (06-02), survive failures with timeouts, retries, circuit breakers, retry queues and the saga watchdog (06-03), scale on their own and with load tests (06-04) and are operated against SLOs with alerts, on-call and postmortems (06-05). The system is now observable, resilient, scalable and operable; what it still is not, apart from the mentions of JWT and data redaction, is secure. Module 7 starts there: authentication and authorization with JWT/OAuth2 and Keycloak (07-01), security in communication between services (07-02), security practices in code and data (07-03) and container and Kubernetes security (07-04).