The Project | About Us | Contribute | Donations | License
HOME My courses Completed
◄ Previous
◄

Failure Management and Recovery

Next ►
►

The previous lesson ended on a red span: inventory-1 isn't responding and orders has opened the circuit towards it. Observability has done its job, which was to point; now the platform has to survive. That means answering questions that never even came up in the monolith: how to tell a node that has crashed from one that is merely slow, who decides that the replica becomes the primary and how you stop the old primary from coming back believing it is still in charge, how to resume a two-hour process that died at minute 90, and how to get the data in km0_inventory back when what failed wasn't a process but a DELETE without a WHERE on a Friday afternoon. This lesson walks the full cycle: detect, tolerate (redundancy and failover with fencing), recover (checkpoints, backups, PITR, RPO/RTO, DR) and manage the incident while all of that is going on. The call patterns between services (timeouts, retries, circuit breakers) are lesson 07-04; orchestration with Kubernetes is 07-05.

Contents

  1. Failures in practice: from the theoretical model to what actually happens
  2. Detection: heartbeats, health checks and false positives
  3. Tolerance through redundancy
  4. Failover, split-brain and fencing
  5. Real-world failover: Patroni, Kafka and Cassandra
  6. State recovery: checkpoints for long-running processes
  7. Data recovery: backups, PITR and restore tests
  8. RPO, RTO and the disaster recovery plan
  9. Incident management: runbooks, on-call and postmortems
  10. Common mistakes and tips
  11. Exercises and solutions
  12. Conclusion

  1. Failures in practice: from the theoretical model to what actually happens

Lesson 01-02 introduced the failure models: crash-stop, crash-recovery, omission, timing and Byzantine. They are useful for reasoning about algorithms; in operations, what you see is their concrete incarnation, and it is almost never the clean case:

What happens Model from 01-02 Example at Kilometre Zero Why it is treacherous
Node crash Crash-stop / crash-recovery The inventory-1 container is killed by OOM This is the "easy" case: it is detected reliably and there is a replica
Network partition Omission The switch between the bcn and vlc zones drops packets for 40 s Each side believes the other has died; both are still alive (03-02)
Full disk Crash preceded by degradation Broker kafka-2 fills its disk with delivery.positions and no retention Before dying it writes slowly: timeouts in producers, lag in consumers
Grey failure (slow but alive) Timing orders-db-primary responds, but every query takes 4 s because of a degraded disk The health checks say "OK"; clients exhaust their pools waiting (07-04)
Correlated failures Several crashes at once A change to the Vault CA leaves all three orders replicas without a valid certificate in the same minute Redundancy doesn't protect you: they share the cause
Human error Any DELETE FROM stock WHERE producer_id = ... with no transaction and the wrong WHERE It replicates perfectly to inv-bcn and inv-vlc: a replica is not a backup
Faulty deployment Crash or grey, correlated A version of orders with a bug that returns 500 in 8% of cases (the Saturday from 07-01) The rolling update carries it to every replica; a rollback is needed (07-05)

Two lessons keep coming back: slow failures are worse than clean failures (a dead node gets replaced; a slow one spreads), and redundancy only protects against independent failures. Everything that follows tries to turn grey failures into clean ones (detect them and take the node out) and to break the correlations (zones, staggered versions, backups outside the system).

  1. Detection: heartbeats, health checks and false positives

A system cannot tolerate a failure it doesn't detect. Detection runs in two directions:

  • Heartbeats: the monitored node periodically sends "I'm alive" (to a coordinator, to its peers, to etcd by renewing a lease). If it stops arriving, the node is presumed dead. This is what Raft (03-03), Cassandra (gossip), Kafka (brokers with the controller) and Patroni (with etcd) use.
  • Health checks: the watcher asks the watched node "are you all right?". This is what Kubernetes, load balancers and Kong do with their upstreams.

Three kinds of health check

The distinction, popularised by Kubernetes (07-05), applies to any platform:

Probe Question If it fails What it should check
Liveness (/health/live) "Is the process hopelessly stuck?" Restart the process Only that the process responds: a blocked internal loop, exhausted memory. Never external dependencies
Readiness (/health/ready) "Can you serve traffic right now?" Take it out of the load balancer without restarting it Essential dependencies: connection to km0_orders, Kafka producer connected, configuration loaded
Startup "Have you finished starting up?" Wait (don't apply liveness yet) Migrations, initial caches, loading certificates from Vault

The classic mistake is checking PostgreSQL in the liveness probe: if the database goes down, every orders replica fails the probe, they restart in a loop, and when PostgreSQL comes back there is nobody left to serve it. With readiness, on the other hand, the replicas withdraw from the load balancer, stay alive, and come back on their own as soon as the dependency responds.

# km0/services/orders/health.py
"""Health checks for orders: liveness without dependencies, readiness with them."""
import asyncio
import time

from fastapi import APIRouter, Response

router = APIRouter(prefix="/health")
STARTED_AT = time.monotonic()


@router.get("/live")
async def live():
    # Liveness: if we can run this function, the process isn't hung.
    # Always return 200; the orchestrator will restart it if it doesn't even respond.
    return {"status": "live", "uptime_seconds": int(time.monotonic() - STARTED_AT)}


async def _check_postgres(pool) -> tuple[bool, str]:
    try:
        async with pool.acquire(timeout=1.0) as conn:        # wait at most 1 s for a connection
            await asyncio.wait_for(conn.execute("SELECT 1"), timeout=1.0)
        return True, "ok"
    except Exception as e:                                    # timeout, connection refused...
        return False, f"postgres: {type(e).__name__}"


async def _check_kafka(producer) -> tuple[bool, str]:
    try:
        # list_topics with a timeout: if the cluster doesn't respond, it raises
        await asyncio.get_running_loop().run_in_executor(
            None, lambda: producer.list_topics(topic="orders.events", timeout=1.0))
        return True, "ok"
    except Exception as e:
        return False, f"kafka: {type(e).__name__}"


@router.get("/ready")
async def ready(response: Response):
    from services.orders.app import pg_pool, kafka_producer   # the service's resources
    results = await asyncio.gather(_check_postgres(pg_pool), _check_kafka(kafka_producer))
    detail = {"postgres": results[0][1], "kafka": results[1][1]}
    if all(ok for ok, _ in results):
        return {"status": "ready", **detail}
    response.status_code = 503        # the load balancer stops sending us traffic
    return {"status": "not_ready", **detail}

The 1 s timeouts on each check are part of the design: a health check without a timeout that sits waiting on a grey PostgreSQL makes the probe itself slow, and the orchestrator reads that as a failure. With timeouts, a grey PostgreSQL turns into "not ready" within a second, which is exactly the conversion from grey failure to clean failure we were after.

Detection timeouts and false positives

Every time-based detector carries a tension: a short threshold detects quickly but declares live nodes dead when they were merely slow (a false positive); a long one is slow to react. On a partitioned network, moreover, it is impossible to tell "dead" from "unreachable" (03-02), so the right question isn't "is it dead?" but "how confident am I that it is?". The phi-accrual detector (Hayashibara et al., used by Cassandra and Akka) answers with a number: instead of a fixed threshold, it learns the distribution of the intervals between heartbeats and computes the suspicion φ as "how unlikely is this delay given what I have seen". A simplified version:

# km0/simulations/heartbeat_detector.py
"""Simplified phi-accrual failure detector.

Each node sends heartbeats; the detector learns the mean and deviation of the
intervals and expresses suspicion as phi = -log10(P(delay >= observed)).
phi = 1 -> 10% chance that this is a normal delay; phi = 3 -> 0.1%.
"""
import math
import statistics
import time
from collections import deque


class PhiDetector:
    def __init__(self, phi_threshold: float = 8.0, window: int = 100, initial_interval: float = 1.0):
        self.threshold = phi_threshold
        self.intervals: dict[str, deque] = {}
        self.last: dict[str, float] = {}
        self.window = window
        self.initial_interval = initial_interval

    def heartbeat(self, node: str, now: float | None = None) -> None:
        now = now or time.monotonic()
        if node in self.last:
            self.intervals.setdefault(node, deque(maxlen=self.window)).append(now - self.last[node])
        self.last[node] = now

    def phi(self, node: str, now: float | None = None) -> float:
        now = now or time.monotonic()
        if node not in self.last:
            return 0.0
        samples = self.intervals.get(node)
        if not samples or len(samples) < 2:
            mean, stdev = self.initial_interval, self.initial_interval / 4
        else:
            mean = statistics.mean(samples)
            stdev = max(statistics.pstdev(samples), mean * 0.05)   # avoid a deviation of 0
        delay = now - self.last[node]
        # Normal-distribution approximation: P(X >= delay)
        z = (delay - mean) / stdev
        p = 0.5 * math.erfc(z / math.sqrt(2))
        return -math.log10(max(p, 1e-12))     # floor so we never divide by 0

    def suspected(self, node: str, now: float | None = None) -> bool:
        return self.phi(node, now) > self.threshold


if __name__ == "__main__":
    # Simulation: inv-bcn sends heartbeats every 1 s with jitter; after 30 s it goes silent.
    import random
    det = PhiDetector(phi_threshold=8.0)
    t = 0.0
    for i in range(30):
        det.heartbeat("inv-bcn", now=t)
        t += 1.0 + random.gauss(0, 0.05)
    last_hb = t
    for delay in (0.5, 1.0, 1.2, 1.5, 2.0, 3.0, 5.0):
        now = last_hb + delay
        print(f"delay {delay:4.1f} s  phi = {det.phi('inv-bcn', now):6.2f}  "
              f"{'SUSPECTED' if det.suspected('inv-bcn', now) else 'alive'}")

Typical output:

delay  0.5 s  phi =   0.00  alive
delay  1.0 s  phi =   0.30  alive
delay  1.2 s  phi =   3.90  alive
delay  1.5 s  phi =  12.00  SUSPECTED
delay  2.0 s  phi =  12.00  SUSPECTED
...

On a very stable network (a deviation of 50 ms), 1.5 s of silence is already highly suspicious; on a network with 300 ms of jitter, the same detector would wait longer before suspecting, without changing a single parameter. That is the value of an adaptive detector: the threshold adjusts to the observed behaviour, and the detector's consumer (whoever decides on the failover) chooses how much confidence to demand. Gossip is the way this state is propagated among many nodes without a coordinator: each node tells a few others, chosen at random, what it knows about the rest (Cassandra uses it for membership and state). Knowing it exists is enough for now.

  1. Tolerance through redundancy

Once the failure is detected, tolerating it requires having something to replace what failed:

Scheme What it is Cost Example at Kilometre Zero
N+1 Capacity for the load with N units, plus one spare 1/N extra orders with 3 replicas when 2 are enough for Grape Harvest Week
Active-active All units serve traffic at the same time Everything works; they have to be interchangeable (stateless, or with replicated state) Replicas of orders, catalog, inventory behind Kong; Cassandra nodes
Active-passive One unit serves; another waits, ready to take over The passive one sits idle; there is a moment of switchover orders-db-primary / orders-db-replica; the outbox relay leader with a lease in etcd
Availability zones Units spread across independent failure domains (building, power, network) Latency between zones; the cost of duplicating inv-bcn in one zone, inv-vlc in another; Kafka brokers spread with rack awareness

What makes redundancy useful is independence: three orders replicas on the same machine don't tolerate that machine going down; three Cassandra nodes in the same rack don't tolerate the switch going down. And, as the failure table showed, three replicas with the same expired certificate tolerate nothing at all. Redundancy is designed per failure domain: machine, rack, zone, software version, credential.

  1. Failover, split-brain and fencing

Failover is the switch from a failed component to its substitute. For stateless services it is trivial: the load balancer stops sending to the not-ready replica (readiness) and that's that. For stateful components with a single writer (the PostgreSQL primary, the leader of a Kafka partition, the outbox relay leader) it is the most delicate problem in the whole lesson, and 03-04 already announced it: split-brain.

The scenario: inv-bcn is the primary and inv-vlc its streaming replica. A 40 s network partition separates inv-bcn from everything else. The detector, on the other side, gives it up for dead and promotes inv-vlc. But inv-bcn isn't dead: it keeps accepting writes from the clients that can still reach it. When the network comes back, there are two primaries with diverging histories: two reservations of the last aged-cheese in two databases that can no longer be reconciled automatically.

The defences, in order of strength:

  1. Consensus for the decision: the one who promotes isn't "whoever sees it dead" but a majority, through a consensus store (etcd, 03-03). At most one node can hold the "leader" lease at a time.
  2. Fencing: guaranteeing that the old primary cannot write before the new one starts. There are several ways, and they combine:
    • Fencing token: each leader receives a monotonically increasing number (the revision of the etcd lease, as in 03-03). Any shared resource (the storage, the outbox consumer) rejects writes carrying a token lower than the last one it saw. The old leader, holding token 41, cannot write where 42 has already been seen.
    • Self-fencing: the leader demotes itself if it fails to renew its lease (that is why the outbox relay checks its lease before each batch and stops if it no longer holds it). It relies on the leader's clock not drifting too far (01-05).
    • STONITH ("shoot the other node in the head"): physically powering off the old node (via IPMI, via the cloud API) before promoting. Brutal, but definitive.
  3. Generous timeouts and for on the decision: don't promote over a 3 s blip.
sequenceDiagram
    participant C as Clients (inventory)
    participant A as inv-bcn (primary, lease rev 41)
    participant E as etcd (3 nodes)
    participant B as inv-vlc (replica)
    Note over A,E: Partition: inv-bcn cannot reach etcd
    A--xE: renew lease (fails)
    Note over A: TTL expired without renewal:<br/>SELF-FENCING -> read-only mode
    E->>B: leader lease expired
    B->>E: acquire lease (rev 42)
    E-->>B: granted
    Note over B: promote: pg_promote()
    B->>C: "I am primary, token 42"
    C->>B: writes with token 42
    Note over A,E: The network comes back
    A->>C: write with token 41
    C-->>A: rejected (41 < 42)
    A->>E: who is the leader?
    E-->>A: inv-vlc (42)
    Note over A: rejoins as a replica<br/>(pg_rewind if it diverged)

Failback (returning to the original node once it recovers) is rarely worth doing automatically: every switch is a risk. The usual practice is for the recovered node to join as a replica and stay that way until a planned switchover.

  1. Real-world failover: Patroni, Kafka and Cassandra

5.1 Patroni for PostgreSQL

Patroni is an agent that runs alongside each PostgreSQL and does exactly what was described: it uses etcd (or Consul, or the Kubernetes API) as the consensus store, holds a leader lease with a TTL, promotes the most advanced replica when the lease expires, and demotes the old primary that fails to renew. For km0_inventory, the docker-compose.yml:

# km0/docker-compose.yml (excerpt): Patroni with 3 nodes for km0_inventory
  etcd-1:
    image: quay.io/coreos/etcd:v3.5.15
    command: ["etcd", "--name=etcd-1", "--initial-cluster=etcd-1=http://etcd-1:2380,etcd-2=http://etcd-2:2380,etcd-3=http://etcd-3:2380",
              "--listen-peer-urls=http://0.0.0.0:2380", "--listen-client-urls=http://0.0.0.0:2379",
              "--advertise-client-urls=http://etcd-1:2379", "--initial-advertise-peer-urls=http://etcd-1:2380"]
  # etcd-2 and etcd-3 are identical apart from the name

  inv-bcn: &patroni
    image: ghcr.io/zalando/spilo-16:3.3-p1      # PostgreSQL 16 + Patroni
    environment: &patroni_env
      SCOPE: km0-inventory                       # cluster name in etcd
      ETCD3_HOSTS: "etcd-1:2379,etcd-2:2379,etcd-3:2379"
      PGPASSWORD_SUPERUSER_FILE: /run/secrets/pg_super
      PATRONI_TTL: "30"                          # leader lease: 30 s
      PATRONI_LOOP_WAIT: "10"                    # renews every 10 s
      PATRONI_RETRY_TIMEOUT: "10"
      PATRONI_MAXIMUM_LAG_ON_FAILOVER: "1048576" # never promote a replica more than 1 MB behind
      PATRONI_SYNCHRONOUS_MODE: "true"           # at least one synchronous replica: RPO 0 on failover
    hostname: inv-bcn
    volumes: ["inv-bcn-data:/home/postgres/pgdata"]
  inv-vlc:
    <<: *patroni
    hostname: inv-vlc
    volumes: ["inv-vlc-data:/home/postgres/pgdata"]
  inv-gir:
    <<: *patroni
    hostname: inv-gir
    volumes: ["inv-gir-data:/home/postgres/pgdata"]

  # Clients don't know who the primary is: HAProxy asks Patroni (/primary returns 200 only on the leader)
  inventory-db:
    image: haproxy:2.9
    volumes: ["./observability/haproxy-patroni.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro"]
    ports: ["5432:5432", "5433:5433"]     # 5432 -> primary (writes); 5433 -> replicas (reads)

The parameters that matter: TTL and LOOP_WAIT define the detection window (up to 30 s without renewal = lease lost); MAXIMUM_LAG_ON_FAILOVER prevents promoting a replica that is too far behind (it would lose more data than is acceptable); SYNCHRONOUS_MODE makes every commit wait for a replica, so a failover loses no committed transactions (the price is write latency, the same tension as in 03-01). HAProxy queries Patroni's REST endpoint to know where to send, so inventory always connects to inventory-db:5432 and never needs to know who the primary is.

Day-to-day operation:

$ patronictl -c /etc/patroni.yml list
+ Cluster: km0-inventory ------+---------+-----------+----+-----------+
| Member  | Host     | Role    | State     | TL | Lag in MB |
+---------+----------+---------+-----------+----+-----------+
| inv-bcn | inv-bcn  | Leader  | running   | 12 |           |
| inv-vlc | inv-vlc  | Sync Standby | streaming | 12 |     0 |
| inv-gir | inv-gir  | Replica | streaming | 12 |         0 |
+---------+----------+---------+-----------+----+-----------+

# Planned switchover (maintenance on inv-bcn): no loss, with confirmation
$ patronictl -c /etc/patroni.yml switchover --leader inv-bcn --candidate inv-vlc --scheduled now
Are you sure you want to switchover cluster km0-inventory, demoting current leader inv-bcn? [y/N]: y
Successfully switched over to "inv-vlc"

# After an unplanned failover, the old leader rejoins; if it diverged, Patroni runs pg_rewind
$ patronictl -c /etc/patroni.yml list
| inv-bcn | inv-bcn  | Replica | streaming | 13 |         0 |
| inv-vlc | inv-vlc  | Leader  | running   | 13 |           |

The TL (timeline) column goes up with every promotion: it is PostgreSQL's "fencing token". A primary on timeline 12 cannot send WAL to replicas that are already on 13.

5.2 Kafka: partition replicas, ISR and leader election

Kafka doesn't have "a primary": each partition has a leader and N-1 follower replicas, spread across brokers. The set of replicas that are up to date is called the ISR (in-sync replicas). When the broker leading a partition goes down, the cluster controller elects a new leader from among the ISR; producers and consumers discover the change on their next metadata request. The parameters that govern what gets lost:

Parameter Value at Kilometre Zero Effect
replication.factor 3 (orders.events, audit.events), 2 (delivery.positions) How many copies of each partition
min.insync.replicas 2 An acks=all is only acknowledged if at least 2 replicas have it; if only 1 ISR is left, the producer gets NotEnoughReplicas (refusing is preferred to losing)
acks (producer) all in orders, 1 in delivery When the producer considers the message stored
unclean.leader.election.enable false Never elect a replica outside the ISR as leader: better an unavailable partition than losing acknowledged messages

The combination replication.factor=3 + min.insync.replicas=2 + acks=all tolerates the loss of one broker without losing a single confirmed order event; with two brokers down, orders.events stops accepting writes (and the outbox from 02-05 holds them until it comes back). delivery.positions accepts losing positions in exchange for latency.

5.3 Cassandra: no leader to fail over

km0_orders on Cassandra (04-04) needs no failover because there is no leader: each row lives on 3 nodes and every read and write with LOCAL_QUORUM needs 2. If one node goes down, operations carry on with the other two; the phi-accrual detector and gossip mark the node as down; when it comes back, hinted handoffs (writes its neighbours kept for it) and repair (nodetool repair) bring it up to date. The price was paid at design time: eventual consistency between replicas and a query-driven data model.

  1. State recovery: checkpoints for long-running processes

Not everything is databases. Every night, inventory runs the stock reconciliation: it walks through the three producers, compares the stock in km0_inventory with the confirmed reservations in km0_orders and the deliveries from delivery, and corrects the deviations. It takes about two hours. If the process dies at minute 90 (OOM, deployment, node crash), starting from scratch means another two hours, and it may not finish before the markets open. The solution is the same idea as Flink's checkpoints (05-04): persist progress periodically somewhere that outlives the process, and resume from the last checkpoint idempotently.

# km0/services/inventory/nightly_reconciliation.py
"""Nightly stock reconciliation with a persisted checkpoint and resumption."""
import json
import time
from dataclasses import dataclass, asdict

import psycopg
from services.common.logs import log

CHECKPOINT_EVERY = 500         # products processed between checkpoints


@dataclass
class Checkpoint:
    run_id: str                # e.g. "2026-09-13"
    current_producer: str      # slug of the producer being processed
    last_product: str          # last confirmed product within that producer ("" = none)
    processed: int
    corrected: int


def load_checkpoint(conn, run_id: str) -> Checkpoint | None:
    row = conn.execute(
        "SELECT state FROM reconciliation_checkpoints WHERE run_id = %s", (run_id,)
    ).fetchone()
    return Checkpoint(**json.loads(row[0])) if row else None


def save_checkpoint(conn, cp: Checkpoint) -> None:
    # UPSERT: the day's row is overwritten; it is committed in the same transaction
    # as the batch's corrections, so there are never corrections without a checkpoint or vice versa.
    conn.execute(
        """INSERT INTO reconciliation_checkpoints (run_id, state, updated_at)
           VALUES (%s, %s, now())
           ON CONFLICT (run_id) DO UPDATE SET state = EXCLUDED.state, updated_at = now()""",
        (cp.run_id, json.dumps(asdict(cp))),
    )


def products_from(conn, producer: str, last_product: str):
    # Deterministic order by slug: resuming means "carry on after the last confirmed one"
    yield from conn.execute(
        "SELECT slug FROM products WHERE producer = %s AND slug > %s ORDER BY slug",
        (producer, last_product),
    )


def reconcile_product(conn, producer: str, product: str) -> bool:
    """Compares stock with reservations and deliveries; corrects it if there is a deviation.
    Idempotent: running it twice on the same product leaves the same result."""
    expected = compute_expected_stock(conn, producer, product)     # queries km0_orders and delivery
    current = conn.execute("SELECT units FROM stock WHERE product = %s FOR UPDATE", (product,)).fetchone()[0]
    if current != expected:
        conn.execute("UPDATE stock SET units = %s WHERE product = %s", (expected, product))
        log.warning("stock_corrected", producer=producer, product=product, before=current, after=expected)
        return True
    return False


def run(run_id: str, producers: list[str]) -> None:
    t0 = time.monotonic()
    with psycopg.connect(INVENTORY_DSN) as conn:
        cp = load_checkpoint(conn, run_id) or Checkpoint(run_id, producers[0], "", 0, 0)
        if cp.processed:
            log.info("reconciliation_resumed", from_producer=cp.current_producer,
                     from_product=cp.last_product, processed=cp.processed)
        # Skip the producers already completed
        for producer in producers[producers.index(cp.current_producer):]:
            start_after = cp.last_product if producer == cp.current_producer else ""
            pending_in_batch = 0
            for (product,) in products_from(conn, producer, start_after):
                if reconcile_product(conn, producer, product):
                    cp.corrected += 1
                cp.processed += 1
                cp.current_producer, cp.last_product = producer, product
                pending_in_batch += 1
                if pending_in_batch >= CHECKPOINT_EVERY:
                    save_checkpoint(conn, cp)
                    conn.commit()            # corrections + checkpoint, atomically
                    pending_in_batch = 0
            save_checkpoint(conn, cp)
            conn.commit()
        log.info("reconciliation_finished", processed=cp.processed, corrected=cp.corrected,
                 duration_s=int(time.monotonic() - t0))


if __name__ == "__main__":
    run(time.strftime("%Y-%m-%d"), ["la-vega-farm", "montblanc-dairy", "roble-alto-winery"])

The three properties that make this work, and that apply to any long-running process:

  1. The checkpoint is committed in the same transaction as the work it represents. If the process dies between the UPDATE and the checkpoint, the transaction isn't committed and both are lost together; you never end up with a checkpoint saying "up to aged-cheese" while aged-cheese is left uncorrected, or the other way round.
  2. The order is deterministic (ORDER BY slug), so "resume from X" has a meaning.
  3. Each unit of work is idempotent: if the checkpoint was saved before a batch of 499 products and the process died, those 499 are reprocessed on resumption, and reprocessing them does no harm.

Airflow (05-05) is what launches this process and relaunches it if it fails (retries): resuming from the checkpoint makes the retry cost minutes rather than hours.

  1. Data recovery: backups, PITR and restore tests

Replication protects against a node going down; it does not protect against bad data. The erroneous DELETE from section 1 reaches inv-vlc in milliseconds. That is what backups are for: copies of the system that are decoupled in time.

Type How Advantages Drawbacks At Kilometre Zero
Logical pg_dump: SQL dump or custom format Portable across versions; restore a single table Slow on large databases; restoring means re-executing; no PITR km0_analytics weekly (can be regenerated from the lake)
Physical pg_basebackup: copy of the data files Fast; the basis for PITR Same major version; all or nothing km0_inventory daily
PITR (point-in-time recovery) Physical backup + continuous WAL archiving Restore to any instant (16:59, one minute before the DELETE) All the WAL since the last base backup must be kept km0_inventory with WAL in km0-backups
Snapshot Copy of Cassandra's SSTables (nodetool snapshot), instant hard links Almost free to create They must be copied off the node; per node km0_orders daily, to km0-backups
Object versioning MinIO keeps previous versions of every object A deletion or overwrite can be undone Storage cost km0-invoices, km0-audit (with object lock, 06-05)

PITR in PostgreSQL step by step

PostgreSQL writes every change first to the WAL (write-ahead log), in 16 MB segments. If every segment since a base backup is kept, history can be replayed up to the desired instant.

flowchart LR
    subgraph normal["Normal operation"]
        PG[(inv-bcn<br/>primary)] -- "archive_command<br/>every WAL segment" --> M[(MinIO<br/>km0-backups/inventory/wal/)]
        PG -- "pg_basebackup<br/>daily at 02:00" --> MB[(km0-backups/inventory/base/2026-09-12/)]
    end
    subgraph recovery["Recovery to 16:59"]
        MB --> R[Restore node]
        M -- "restore_command<br/>replay up to recovery_target_time" --> R
        R --> V{data correct?}
        V -- yes --> P[pg_promote → new primary<br/>Patroni reinitialises replicas]
    end

Archiving configuration on the primary (Patroni applies it under postgresql.parameters):

# km0/sql/backup/archive_wal.sh — invoked by PostgreSQL with %p (path) and %f (segment name)
#!/usr/bin/env bash
set -euo pipefail
# mc: MinIO client; the 'km0' alias was configured with credentials from Vault (06-04)
mc cp --quiet "$1" "km0/km0-backups/inventory/wal/$2"
# PostgreSQL parameters managed by Patroni (excerpt from patroni.yml)
postgresql:
  parameters:
    wal_level: replica
    archive_mode: "on"
    archive_command: "/opt/km0/sql/backup/archive_wal.sh %p %f"
    archive_timeout: 60        # force a segment every minute even if it isn't full: RPO <= 1 min

Daily base backup:

# km0/sql/backup/base_backup.sh
#!/usr/bin/env bash
set -euo pipefail
DATE=$(date -u +%F)
DEST=/backups/base/$DATE
mkdir -p "$DEST"
# -Ft: tar; -z: compressed; -X stream: includes the WAL needed for the backup to be self-consistent
pg_basebackup -h inventory-db -p 5432 -U replicator -D "$DEST" -Ft -z -X stream --checkpoint=fast
mc cp --recursive "$DEST" "km0/km0-backups/inventory/base/$DATE/"
# Keep 14 days of base backups; the WAL older than the oldest one is no longer useful
mc rm --recursive --force --older-than 14d km0/km0-backups/inventory/base/

And the restore to 16:59 on Friday, one minute before the 17:00 DELETE:

# 1. Clean node (inv-rest): download the latest base backup taken BEFORE the target instant
mc cp --recursive km0/km0-backups/inventory/base/2026-09-11/ /restore/base/
mkdir -p /restore/pgdata && cd /restore/pgdata
tar -xzf /restore/base/base.tar.gz
tar -xzf /restore/base/pg_wal.tar.gz -C pg_wal/

# 2. Tell PostgreSQL how far to replay and where to fetch the WAL from
cat >> postgresql.auto.conf <<'EOF'
restore_command = 'mc cp --quiet km0/km0-backups/inventory/wal/%f %p'
recovery_target_time = '2026-09-11 16:59:00+02'
recovery_target_action = 'promote'      # on arrival, leave recovery mode
EOF
touch recovery.signal

# 3. Start it: replays the day's WAL up to 16:59 and promotes itself
pg_ctl -D /restore/pgdata start
# LOG:  starting point-in-time recovery to 2026-09-11 16:59:00+02
# LOG:  restored log file "000000010000004A000000F3" from archive
# ...
# LOG:  recovery stopping before commit of transaction 8812345, time 2026-09-11 16:59:12.8
# LOG:  database system is ready to accept connections

# 4. Verify before touching production
psql -d km0_inventory -c "SELECT count(*), sum(units) FROM stock WHERE producer = 'montblanc-dairy';"

# 5. Decide: (a) extract the deleted rows and reinsert them into production with a script (the usual
#    choice, if production has kept receiving orders after 17:00), or
#    (b) turn inv-rest into the new primary of the Patroni cluster and reinitialise the replicas
#    (if the damage is so severe that losing everything after 16:59 is preferable).

Step 5 is the one textbook exercises forget: between 17:00 and the moment of the restore, km0_inventory has kept receiving reservations. Rolling the whole thing back to 16:59 would lose them. Almost always the choice is to restore on the side and re-inject what was missing.

Cassandra and MinIO

# Snapshot on each km0_orders node (hard links to the current SSTables: instant)
nodetool snapshot -t daily-2026-09-12 km0_orders
# Copy off the node (per keyspace/table) and delete the local snapshot
mc cp --recursive /var/lib/cassandra/data/km0_orders/*/snapshots/daily-2026-09-12/ \
      km0/km0-backups/cassandra/$(hostname)/2026-09-12/
nodetool clearsnapshot -t daily-2026-09-12
# Restore: copy the SSTables into the table's directory and run `nodetool refresh km0_orders orders`
# (or load them with sstableloader into a different cluster)

# MinIO: enable versioning on the buckets where accidental deletion matters
mc version enable km0/km0-invoices
mc undo km0/km0-invoices/2026/09/F-2026-004411.pdf   # undo the last deletion or overwrite

Restore tests

A backup that has never been restored isn't a backup: it is a hope. The usual failures are utterly prosaic: the archive_command has been failing silently for three weeks (alert: pg_stat_archiver.failed_count goes up, and it is the pg_stat_archiver_failed_count metric from postgres_exporter); the base backup is corrupt; the restore needs a parameter nobody remembers; it takes six hours and the RTO was one. That is why Kilometre Zero has an Airflow DAG, km0_restore_test, which every Sunday restores the latest km0_inventory backup into an ephemeral container, runs a few verification queries (number of products per producer, sum of stock) against what was recorded at backup time, measures the time taken, and publishes km0_backup_restore_ok{db="inventory"} and km0_backup_restore_seconds to Prometheus. If it fails, it is a ticket alert with the same priority as a production failure.

  1. RPO, RTO and the disaster recovery plan

Two numbers sum up what a system promises in the face of a disaster:

  • RPO (Recovery Point Objective): how much data, measured in time, you accept losing. An RPO of 1 minute means the last usable copy is at most one minute old.
  • RTO (Recovery Time Objective): how long the service may be down before it is recovered.

Both are set by the business and paid for in architecture: RPO 0 demands synchronous replication; an RTO of minutes demands automatic, tested failover. For Kilometre Zero:

Component RPO RTO How it is achieved What is lost if it is exceeded
km0_inventory (PostgreSQL) 0 on failover (synchronous replica); 1 min with PITR (archive_timeout) 1 min (Patroni); 1 h (PITR) Patroni with 3 nodes; WAL to km0-backups; weekly test Stock reservations: overselling on the producers' behalf
km0_orders (Cassandra, 3 nodes) 0 with one node down; 24 h on total loss (daily snapshot) 0 with one node down; 4 h on total loss RF 3 + LOCAL_QUORUM; snapshots to MinIO; the events in orders.events allow a rebuild Orders: the most direct harm to customers
orders-db-* (PostgreSQL, streaming) seconds (asynchronous replica) 5 min (documented manual promotion) Streaming + runbook Saga state: pending compensations
Kafka (orders.events, audit.events) 0 (RF 3, min.insync.replicas 2, acks=all) 30 s (leader election) Topic configuration Events: incomplete analytics and audit
Kafka (delivery.positions) minutes (RF 2, acks=1) 30 s Accepted: positions are ephemeral Nothing significant
Redis (cache) ∞ (regenerated) 1 min No backup; the catalogue is warmed up again (04-05) A load spike on PostgreSQL
MinIO km0-invoices, km0-audit 0 (versioning, object lock, bucket replication to another region) 1 h Bucket replication Legal obligations
km0_analytics 24 h 8 h Weekly dump + re-running the DAG from the lake Reports: recomputed

The last line of defence is the disaster recovery (DR) plan: what to do if an entire data centre disappears (fire, prolonged power cut, provider error). The strategies, from cheapest to most expensive:

Strategy What exists in the secondary region Typical RPO / RTO Cost
Backup and restore Only the backups (km0-backups replicated) Hours / hours-days Minimal
Pilot light Data replicated live (PostgreSQL replicas, Kafka mirror); services switched off, ready to start Minutes / tens of minutes Low
Warm standby Everything deployed at reduced scale and receiving replication; scaled up on switchover Seconds-minutes / minutes Medium
Active multi-region Both regions serve traffic; data replicated in both directions ~0 / ~0 High, and complex (conflicts, 03-04)

Kilometre Zero opts for pilot light: MinIO bucket replication to the secondary region, one PostgreSQL replica of each cluster in the other region (asynchronous, outside the Patroni quorum) and a mirror of the critical Kafka topics. Where that secondary region lives and how it is deployed with managed services is the subject of 08-03. What does belong to this lesson: the plan is rehearsed (a game day every six months, 07-06), it has a runbook, and the criterion for activating it is written down in advance, because in the middle of a disaster nobody thinks clearly.

  1. Incident management: runbooks, on-call and postmortems

Everything so far has been mechanisms; incidents are managed by people under pressure. Three brief practices:

Runbooks. Every paging alert (07-01) links to a document with: what it means, how to confirm it, what to look at (specific dashboards and queries), actions ordered from least to most invasive, and when to escalate. The runbook for OrdersFastBurnRate begins with "has there been a deployment in the last hour? If so, rollback first, investigate afterwards". They are written calmly, in advance, and corrected after every use.

On-call. Somebody (Jordan or Martha this week) receives the pages, with a secondary as backup, weekly rotation, and compensation. The load is measured: more than two pages a night is a problem with the system, not with the person. An incident has a coordinator (who communicates and decides) separate from whoever is investigating, and a channel with a timestamped record, which will be the raw material for the postmortem.

Blameless postmortems. After every significant incident: timeline, impact (in terms of SLO and error budget), contributing causes (plural: it is almost never just one), what worked, what didn't, and actions with an owner and a date. "Blameless" is not politeness: if the outcome were to point at whoever ran the DELETE, next time nobody would say what happened, and the system that allowed a DELETE without a transaction to run in production would stay exactly as it is. The right action is "operator sessions on km0_inventory start with SET default_transaction_read_only = on" and "PITR tested weekly", not "be more careful".

Common Mistakes and Tips

  • Checking dependencies in the liveness probe. It restarts every replica in a loop when PostgreSQL goes down. Liveness: the process only. Readiness: the dependencies, with timeouts.
  • Health checks without a timeout. A grey dependency makes the probe slow and the orchestrator reads it as a failure, but late and badly. Each check with its own 1 s limit.
  • Promoting without fencing. "We see it down, we promote" is the recipe for split-brain. Consensus to decide (etcd/Patroni), a token or timeline to reject the old one, self-fencing by lease.
  • Believing the replica is a backup. It replicates mistakes just as faithfully as correct data. Physical backup + archived WAL, outside the system, with retention.
  • Backups that are never restored. They are a hope. Automated restore test, measured and alerted on.
  • Restoring everything to a past instant without thinking about what came after. The legitimate transactions since then are lost. Restore on the side and re-inject.
  • RPO/RTO never written down. If they aren't written, the system has whatever they turn out to be. A table per component, agreed with the business, and an architecture that meets it.
  • Redundancy without independence. Three replicas on the same machine, with the same certificate, deployed at the same time. Distinct failure domains along every axis.
  • A postmortem that ends in "be more careful". That isn't an action. Every contributing cause gets a change to the system with an owner and a date.

Exercises

Exercise 1. During Grape Harvest Week, the disk on inv-bcn starts to degrade: it still responds, but every UPDATE takes 3-4 s. Patroni renews the lease without trouble (etcd responds), inventory's /health/ready keeps returning 200 because the SELECT 1 takes 20 ms, and the latency alerts for orders fire. (a) What kind of failure is this and why does neither detector see it? (b) Propose one change to inventory's readiness probe and another to the Patroni configuration or the operating procedure that would turn this grey failure into a clean switchover, and comment on the false-positive risk of each. (c) After the switchover to inv-vlc, what happens to the transactions inv-bcn had in flight, and what guarantees that no reservation confirmed to orders is lost?

Exercise 2. On Friday at 17:00 an operator runs UPDATE stock SET units = 0 WHERE producer = 'roble-alto-winery' on km0_inventory, believing they are in the test environment. It is detected at 17:25 by the business alert "reservations rejected for stock" (07-01). The latest base backup is from 02:00 and the WAL is archived every minute. (a) Describe the complete procedure, with the commands from section 7, to recover Roble Alto Winery's stock without losing the reservations that other producers received between 17:00 and 17:25. (b) Roughly how much WAL has to be replayed, and what does the real RTO depend on? (c) Write two postmortem actions that aren't "be more careful".

Exercise 3. The nightly reconciliation crashes at 03:40 after 95 minutes, with the checkpoint {"current_producer": "montblanc-dairy", "last_product": "fresh-cheese", "processed": 1150}. Airflow relaunches it at 03:42. (a) Exactly where does it carry on from and which products get reprocessed? (b) A colleague proposes storing the checkpoint in Redis "because it's faster". What is lost? (c) Another proposes that reconcile_product send an event to inventory.alerts every time it corrects stock; what problem appears on resumption, and how would you solve it with what was covered in 02-05?

Solutions

Exercise 1.

(a) It is a grey failure (timing model): the node is alive and answers the lightweight checks (Patroni's lease, SELECT 1), but it is useless for real work. The detectors measure "does it respond", not "does it respond to what matters". (b) inventory's readiness: replace SELECT 1 with a representative check under the same 1 s timeout (for example, UPDATE probe SET ts = now() WHERE id = 1, a dedicated row that exercises the WAL and the disk); if it takes longer than 1 s, 503 and the inventory replicas leave the load balancer, which at least stops the contagion spreading to orders (the circuit from 07-04 will do the rest); risk: a one-off I/O spike takes every replica out at once, so you need for/a threshold of consecutive failures (3 in a row) before withdrawing. In Patroni: there is no built-in disk-latency detection, so the route is an alert on pg_stat_*/I/O latency from node_exporter (USE for the disk, 07-01) with a runbook whose first step is patronictl switchover --leader inv-bcn --candidate inv-vlc; or, if you want it automatic, a custom watchdog that runs that switchover when write latency exceeds N seconds for M minutes, with the risk of switching over because of legitimately high load; switching over is cheaper than overselling, so conservative thresholds, but automated. (c) Patroni's switchover performs a checkpoint, waits for the synchronous replica to be up to date and demotes inv-bcn; in-flight (uncommitted) transactions are aborted and the client receives an error, which orders translates to UNAVAILABLE and retries against the new primary (07-04, provided the reservation is idempotent, 02-05); no committed transaction is lost because SYNCHRONOUS_MODE guaranteed the commit was already on inv-vlc before replying to orders.

Exercise 2.

(a) 1) Confirm the scope: in production, SELECT count(*) FROM stock WHERE producer='roble-alto-winery' AND units=0 and the operator's audit log (06-05) to pin down the exact time (17:00:12). 2) On a separate node (inv-rest), restore the 02:00 base backup and replay WAL with recovery_target_time = '2026-09-11 17:00:00+02' (just before the UPDATE), recovery_target_action = 'promote'. 3) Verify on inv-rest: SELECT product, units FROM stock WHERE producer='roble-alto-winery'. 4) Compute the correct current stock: the 17:00 value on inv-rest minus the reservations confirmed between 17:00 and 17:25 in production (which are valid: some failed because of stock 0, but the ones that got in before 17:00:12 or belong to other producers are left untouched); in practice, for Roble Alto Winery: units_17:00 - confirmed_reservations_since_17:00(product), obtained from km0_orders/orders.events. 5) Apply in production with an UPDATE ... FROM (VALUES ...) inside a transaction, compare before COMMIT, and record the correction. 6) Re-run the reconciliation from section 6 for that producer as verification. Never turn inv-rest into the primary: 25 minutes of reservations from La Vega Farm and Montblanc Dairy would be lost. (b) 15 hours of WAL (02:00 → 17:00); on a normal km0_inventory day it can run to tens of GB during the Harvest; the real RTO depends on download speed from MinIO and replay speed (single-threaded in PostgreSQL), on whether the runbook has been tested (the Sunday DAG gives the figure: if it says 40 min, a 1 h RTO is credible) and on the time to compute and apply the correction. (c) "Operator connections to production open with default_transaction_read_only = on and a separate role that requires an explicit, audited SET ROLE writer"; "the psql prompt and the production host name carry PROD and a different colour, and test environments don't share Vault credentials with production"; "business alert reservations rejected for stock with for: 2m instead of 15, which would have caught it at 17:03".

Exercise 3.

(a) It starts at montblanc-dairy, with products_from(conn, "montblanc-dairy", "fresh-cheese"), that is, the first product whose slug sorts after fresh-cheese alphabetically; La Vega Farm is not touched (already completed). Since the checkpoint is only committed every 500 products or when a producer finishes, the products processed between the last checkpoint and the crash (up to 499) are reprocessed; that is correct because reconcile_product is idempotent. (b) You lose the atomicity between checkpoint and corrections: Redis and PostgreSQL don't share a transaction, so you can end up with the checkpoint advanced while the corrections were never committed (products would be skipped without reconciliation) or the other way round (they are merely reprocessed, which is harmless); in addition, Redis without persistence can lose the checkpoint on a restart. Speed is irrelevant: one row is written every 500 products. (c) On resumption, the reprocessed products that were already corrected in the committed transaction won't be corrected again (current already equals expected), but those in the uncommitted batch are corrected again... and in fact the problem is the opposite one: if the event is sent to Kafka directly from reconcile_product, it is published before the commit, and if the process dies there is an event in inventory.alerts for a correction that never happened; or, if it is retried, a duplicate event. The solution is the Outbox pattern from 02-05: the correction and the outbox row are written in the same transaction as the checkpoint, and the relay publishes them afterwards; consumers deduplicate by key (product + run_id).

Conclusion

Detect, tolerate, recover. The platform detects with heartbeats and with health checks separated by intent (liveness without dependencies, readiness with them and with timeouts, startup for booting), and with adaptive detectors that express suspicion rather than certainty, because on a network you cannot tell dead from unreachable. It tolerates with redundancy that is only worth anything if it is independent per failure domain, and with failover that is only safe if a majority decides and the old primary is fenced off: Patroni over etcd for km0_inventory (lease, timeline, synchronous replica, switchover), ISR and min.insync.replicas for Kafka, and leaderless quorum for Cassandra. It recovers state with checkpoints committed in the same transaction as the work, and data with physical backups and WAL archived in km0-backups that allow going back to the minute before the mistake, Cassandra snapshots and MinIO versioning, all of it useless unless it is genuinely restored every week. RPO and RTO put numbers, per component, on what is promised; the pilot light DR plan covers the loss of a region; and runbooks, on-call and blameless postmortems turn every incident into a change to the system rather than a reproach.

There is still a gap between detection and failover: those 30 seconds during which inv-bcn isn't responding and Patroni hasn't yet promoted inv-vlc, or the four seconds of every UPDATE in the grey failure from exercise 1. During that interval, orders keeps calling inventory, each call ties up a thread waiting, the threads run out, Kong starts queueing, and a problem with one disk turns into an outage of the whole platform. Tolerating failures isn't only about replacing what fails; it is about making sure the caller of what fails doesn't sink with it. Those are the resilience patterns of the next lesson: timeouts with a budget, retries that don't make things worse, the circuit breaker that 01-04 left pending, bulkheads, graceful degradation and load shedding.

◄ Previous
◄
Next ►
►

Distributed Architectures Course

Module 1: Introduction to Distributed Systems

Module 2: Communication in Distributed Systems

Module 3: Consistency and Replication

Module 4: Distributed Storage

Module 5: Distributed Computing

Module 6: Security in Distributed Systems

Module 7: Monitoring and Maintenance

Module 8: Case Studies and Applications

The Project | About Us | Contribute | Donations | License
© Copyright 2026. All rights reserved