In 01-06 we opened up a Python monolith with a PostgreSQL database, a Tuesday-and-Thursday deployment and five symptoms pushing it to change, and we drew a target architecture with a table that pointed each component at a future lesson. Thirty-eight lessons later, every one of those boxes has been built: gRPC and Kafka, sagas and replicas, Cassandra and Redis, Spark and Flink, Keycloak and Vault, Prometheus and Kubernetes, MQTT and WebSockets, Terraform and Lambda, the CDN and the van's SQLite queue. This last lesson adds no new piece: it assembles the ones we have. First the complete architecture in a single diagram and the table from 01-06 closed off; then the journey of one of Anna's orders from her browser to the invoice in S3, with everything it leaves in its wake in traces, metrics and audit records; then the six architecture decisions as ADRs, with their rejected alternatives and their consequences; the five symptoms of 01-06 with their resolution; the assessment of the platform against the course's own criteria (fallacies, consistency per piece of data, SLOs, RPO/RTO, security controls); what is left out; the project the student can build on their laptop with verifiable milestones; and a guide for going deeper. The exercises are integration exercises: bringing in a new domain, analysing an incident end to end and trimming the architecture down for three people. The conclusion closes the course.

Contents

  1. The final architecture of Kilometre Zero
  2. The table from 01-06, closed
  3. The journey of one of Anna's orders end to end
  4. What the order leaves in its wake: traces, metrics and audit
  5. The architecture decisions as ADRs
  6. The five symptoms of 01-06, revisited
  7. Assessment against the course's criteria
  8. What is left out and next steps
  9. The project: a reduced Kilometre Zero in docker-compose
  10. Study guide
  11. Common Mistakes and Tips
  12. Exercises
  13. Conclusion

  1. The final architecture of Kilometre Zero

flowchart TB
    subgraph Clients
        ANNA["Anna's browser / phone"]
        CAPP[Courier app]
        PDASH[Producer dashboard]
        VAN[van-3<br/>SQLite aggregator]
    end

    subgraph Edge[Network edge]
        CDN[CDN + Worker<br/>cache per market, JWT]
    end

    subgraph Region[Region eu-west-1, 3 AZs]
        ALB[ALB] --> KONG[Kong<br/>jwt, rate-limiting, X-Request-Id]
        KONG --> BFF[Courier app BFF]
        subgraph EKS[EKS + service mesh mTLS]
            CAT[catalog]
            ORD[orders<br/>saga, outbox]
            INV[inventory<br/>gRPC]
            PAY[payments<br/>gateway ACL]
            DEL[delivery<br/>ws_server, MQTT bridge]
            ANL[analytics]
            MQ[Mosquitto]
            OBS[Prometheus · Loki · Tempo · Grafana]
            KC[Keycloak realm km0]
            VLT[Vault / Secrets Manager]
        end
        KONG --> CAT & ORD & DEL
        BFF --> DEL & ORD & CAT
        ORD -- gRPC mTLS --> INV
        ORD -- gRPC mTLS --> PAY
        PAY --> GW[(Payment gateway)]
        subgraph Data
            RDS[(RDS PostgreSQL Multi-AZ<br/>km0_inventory)]
            CAS[(Cassandra 3 nodes<br/>km0_orders)]
            RED[(ElastiCache Redis<br/>cache, rate limiter, pub/sub)]
            S3[(S3 km0-photos, invoices,<br/>backups, audit)]
            LAKE[(S3 lake /km0/events)]
            ANDB[(PostgreSQL km0_analytics)]
        end
        INV --> RDS
        ORD --> CAS
        CAT --> RED
        CAT --> S3
        subgraph Messaging
            MSK[(MSK Kafka<br/>orders.events, inventory.alerts,<br/>delivery.positions, delivery.dashboard,<br/>audit.events)]
        end
        ORD -. outbox .-> MSK
        INV -. outbox .-> MSK
        MSK -.-> CAT & DEL & ANL
        MQ --> DEL --> MSK
        subgraph Compute
            FLK[Flink stock_alerts, dashboard]
            SPK[Spark daily_sales, EMR spot]
            AIR[Airflow km0_daily_sales]
        end
        MSK --> FLK --> MSK
        MSK --> LAKE --> SPK --> ANDB
        AIR --> SPK
        subgraph Serverless
            LTHUMB[Lambda thumbnails]
            LINV[Lambda invoices]
        end
        S3 -- s3:ObjectCreated --> LTHUMB --> S3
        MSK -- payment.confirmed --> LINV --> S3
    end

    subgraph DR[Region eu-central-1, pilot light]
        RDSR[(RDS replica)]
        S3R[(Replicated S3)]
        MSKR[(Minimal MSK, MirrorMaker)]
    end
    RDS -.-> RDSR
    S3 -.-> S3R
    MSK -.-> MSKR

    ANNA --> CDN --> ALB
    PDASH --> CDN
    CAPP --> ALB
    VAN -- MQTT QoS 1 --> MQ
    DEL -- WebSocket --> ANNA
    KC -. OIDC .- ANNA & CAPP & PDASH

The diagram has one piece that until now was only promised: the courier app BFF, announced in 06-05 and placed in 08-01. It is a small service, owned by the delivery team, behind Kong, that composes the app's home screen (the day's route from delivery, the status of that route's orders from orders, product names and photos from catalog) into a single response, with parallel calls, short timeouts and graceful degradation (if catalog does not answer, the route is shown without photos). It has no database: it is API composition (08-01, section 5), justified because the app makes few calls, the data is small and it needs freshness.

  1. The table from 01-06, closed

The table in 01-06 was a promise: each component, and the lesson where it would be built. This is the same table, closed, with what was finally built and the file or name that materialises it in km0/.

Component Lesson What was built
Protocols: TCP/UDP/HTTP; MQTT and WebSockets introduced 02-01 TCP client with timeout; header measurement
Synchronous RPC between services 02-02 Stubs, marshalling, failure semantics
gRPC with contracts 02-03 contracts/inventory.proto (ReserveStock, GetStock, WatchChanges)
Event bus and queues 02-04 Kafka orders.events (6 partitions, key = order), envelope event_id/type/version/timestamp_ms/source/data
Idempotency, outbox, DLQ, competing consumers 02-05 outbox table, processed_messages, .retry topics and DLQ
Consistency models; CRDTs 03-01 Consistency-per-data table; G-Counter
CAP and PACELC: inventory (C) and orders (A) 03-02 Decision per domain
Consensus: Raft, etcd, leader election 03-03 etcd for Patroni; Kafka controller
Replication: primary/replica, multi-leader, quorums 03-04 inv-bcn/inv-vlc; Patroni
Sagas 03-05 order_saga.py, sagas table, orchestration and choreography
Partitioning and consistent hashing 04-01 Ring, Kafka and Cassandra partitions
Distributed file systems 04-02 Lake /km0/events/... (HDFS, later S3)
Object storage 04-03 MinIO/S3 km0-photos, km0-invoices, km0-backups, km0-audit; photos.py; presigned URLs
Distributed databases 04-04 Cassandra km0_orders (orders_by_customer, orders_by_id), repository with consistency levels
Caches 04-05 Redis cache-aside for catalog, invalidation on stock.updated, Redis Cluster
Computing models 05-01 BSP, dataflow, actors
MapReduce and Hadoop 05-02 Sales per market over the lake
Spark 05-03 daily_sales.py
Streams 05-04 Flink stock_alerts.py (inventory.alerts), dashboard delivery.positionsdelivery.dashboard
Pipelines 05-05 Airflow DAG km0_daily_sales
AuthN/AuthZ 06-01 JWT (sub, roles, jti), RBAC/ABAC in services
Encryption 06-02 TLS, encryption at rest with our own keys, field-level encryption for payments and positions
Identities 06-03 Keycloak realm km0, clients web-km0, courier-app, orders-service; OIDC
mTLS and secrets 06-04 km0-ca, AuthInterceptor/ServiceInterceptor, Vault (AppRole, dynamic credentials)
Gateway and auditing 06-05 Kong (/api/v1/catalog, /api/v1/orders, jwt, rate-limiting, X-Request-Id), audit.events, km0-audit
Metrics and SLOs 07-01 services/common/metrics.py, km0_orders_total, SLO 99.9% and p99 < 500 ms, error budget burn rate alerts
Logs and traces 07-02 services/common/{logs,traces}.py, Loki, Tempo, OpenTelemetry, X-Request-Id
Failures and recovery 07-03 Patroni + etcd, ISR, checkpoints, backups and PITR in km0-backups, RPO/RTO, pilot light DR, runbooks
Resilience 07-04 services/common/resilience.py: timeouts, retries with jitter, circuit breaker, bulkhead, load shedding
Automation and orchestration 07-05 Ansible, k8s/orders-deployment.yaml, HPA, canary, GitOps, mesh
Testing and chaos 07-06 Testcontainers, contracts, k6 tests/load/harvest.js, Toxiproxy, Chaos Mesh
Microservices 08-01 Bounded contexts, services/inventory/api.py, products_view (CQRS), versioning /v1/v2, ADR-001
Real time 08-02 Mosquitto with ACL, van_mqtt.py, mqtt_kafka_bridge.py, ws_server.py, tracking.js, SSE alerts
Cloud 08-03 infra/aws/main.tf (VPC across 3 AZs, EKS, RDS Multi-AZ, MSK, ElastiCache, S3, IRSA), FinOps
Serverless and edge 08-04 serverless/thumbnails, serverless/invoices, template.yaml, saga in Step Functions, CDN Worker, edge/van/aggregator.py
End to end 08-05 This lesson

  1. The journey of one of Anna's orders end to end

Anna, in Valencia, buys two aged-cheese from Montblanc Dairy and one crianza-wine from Roble Alto Winery. The order is P-2026-000125. This is the complete journey, with the lesson in which each step was built.

sequenceDiagram
    autonumber
    participant A as Anna's browser
    participant CDN as CDN + Worker
    participant K as Kong
    participant P as orders
    participant I as inventory
    participant PG as payments
    participant KF as Kafka
    participant C as catalog
    participant AN as analytics (Flink, lake)
    participant R as delivery
    participant L as Lambda invoices

    A->>CDN: GET /api/v1/catalog/products/aged-cheese (JWT)
    CDN-->>A: HIT: listing for market valencia (08-04)
    A->>CDN: POST /api/v1/orders (JWT, Idempotency-Key)
    CDN->>K: verifies JWT signature, forwards (not cacheable)
    K->>K: jwt plugin, rate-limiting, X-Request-Id (06-05)
    K->>P: POST /orders + X-Request-Id
    P->>P: Idempotency-Key seen before? no → create order (created) in Cassandra (04-04)
    P->>P: start saga: row in sagas (03-05)
    P->>I: gRPC ReserveStock (mTLS, ServiceInterceptor, 300 ms timeout, circuit breaker)
    I->>I: SELECT FOR UPDATE on RDS, 15 min reservation; outbox: stock.reserved, stock.updated
    I-->>P: Reservation OK
    P->>PG: gRPC Charge (Idempotency-Key charge-P-2026-000125)
    PG->>PG: external gateway (ACL), 1.2 s
    PG-->>P: payment authorized
    P->>P: status paid; outbox: order.created, payment.confirmed (same transaction)
    P-->>K: 201 Created {id, status: paid}
    K-->>CDN: 201 Created
    CDN-->>A: 201 (p99 < 500 ms excluding the gateway: SLO 07-01)
    Note over P,KF: outbox relay → orders.events (partitioned by order id)
    KF-->>C: stock.updated → products_view + invalidate Redis (08-01, 04-05)
    KF-->>AN: order.created, payment.confirmed → lake /km0/events; Flink stock_alerts (05-04)
    KF-->>R: payment.confirmed → assign van-3, publish delivery.assigned
    KF-->>L: payment.confirmed (group invoices-lambda) → PDF in km0-invoices (08-04)
    A->>R: wss://.../ws {jwt}, {subscribe: P-2026-000125}
    R->>R: authorize sub = customer; channel van:van-3
    loop every 5 s
        Note over R: van-3 → MQTT → bridge → delivery.positions → Redis pub/sub
        R-->>A: {type: position, seq, lat, lon}
    end

Five observations about the diagram that sum up the course:

  1. Only two synchronous calls between services (steps 9 and 12), both with a timeout, a circuit breaker and mTLS, and both because their response governs the next decision (08-01). Everything else is events.
  2. No transaction crosses services. The order is created in Cassandra, the reservation in RDS and the charge at the gateway as local transactions; the saga chains and compensates them (03-05); the outbox guarantees that events go out if and only if the local transaction commits (02-05).
  3. Idempotency appears four times: Anna's Idempotency-Key at Kong/orders (a double click does not create two orders), the reservation_id in inventory, the charge-<order> key towards the gateway, and the event_id in every consumer. A retry at any point is safe.
  4. Anna receives her response at step 17, before the catalogue view, the delivery assignment or the invoice exist. What she sees afterwards (the van's position, the invoice in her account area) arrives by asynchronous paths with seconds of delay, which is the normal state (08-01).
  5. The edge and the gateway do their job before anything reaches the services: the JWT is verified three times (Worker, Kong, service), rate limiting protects orders, and the product listing never even reached the region.

  1. What the order leaves in its wake: traces, metrics and audit

An order does not only produce state; it produces evidence. What remains after P-2026-000125:

Signal Where Content Lesson
Trace Tempo A trace_id generated at Kong (or propagated from the browser), with spans: kongorders POST /ordersinventory.ReserveStock (12 ms) → payments.Charge (1,210 ms, with child span gateway) → cassandra.insertoutbox.insert. Afterwards, linked spans (not children) in the consumers: catalog.projection, delivery.assign, invoices.handler, joined by the event_id propagated in the Kafka headers 07-02
Logs Loki JSON lines with request_id, trace_id, order_id, sub=u-anna, level and message, in every service; queryable with {service="orders"} | json | order_id="P-2026-000125" 07-02
Metrics Prometheus km0_orders_total{status="confirmed"} +1; latency histogram for POST /orders (0.5 s bucket); km0_grpc_client_duration_seconds{method="ReserveStock"}; km0_circuit_state{dependency="payments"}=0; lag of the catalog-projection group; km0_ws_connections +1 07-01
Audit audit.eventskm0-audit (immutable, with legal retention) {"sub": "u-anna", "action": "orders.create", "resource": "P-2026-000125", "remote": ..., "request_id": ..., "result": "ok"}; and the access of orders-service to inventory recorded by the ServiceInterceptor 06-05
Business events orders.events (7-day retention) and lake /km0/events/orders.events/date=2026-09-15/ (forever) order.created, stock.reserved, stock.updated, payment.confirmed, delivery.assigned 02-04, 04-02
State Cassandra (orders_by_customer, orders_by_id, sagas), RDS (skus, reservations, outbox), products_view, S3 (invoices/u-anna/P-2026-000125.pdf) The order, the reservation, the completed saga, the view and the invoice Modules 3 and 4
Cost Provider bill, by tag A few fractions of a cent: invocations, transitions, GB-s, invoice egress 08-03

The proof that observability has been done well is that with the order_id everything can be reconstructed: from the logs to the trace_id, from the trace to each service and its latency, from the metric to whether that order was one of those that broke the SLO, from the event in the lake to what analytics computed, from the audit record to who did it. That is what exercise 2 puts to the test.

  1. The architecture decisions as ADRs

In 08-01 the full ADR-001 was written. These are Kilometre Zero's six structural decisions in summary format: each with the alternative that was rejected and the negative consequence that was accepted, which is the part that in a year's time will tell whether it is still valid.

ADR Decision Rejected alternative and why Accepted consequence Lesson
ADR-001 Extract inventory from the monolith as a bounded context with km0_inventory (PostgreSQL with Patroni / RDS Multi-AZ) and a gRPC API + events Keep stock inside orders (would couple rules and deployments); stock in Cassandra (lightweight transactions expensive for compare-and-set); 2PC (locks and an external gateway outside the protocol) Eventual consistency between real stock and catalogue (a window of seconds); one more synchronous call on the critical path (+12 ms p99); a stateful component to operate 08-01
ADR-002 Cassandra (3 nodes, RF=3, QUORUM) for km0_orders, with one table per query PostgreSQL with Citus (a coordinator and distributed transactions that the access pattern does not need); MongoDB (single primary with failover); PostgreSQL and accepting 30 s of unavailability during a campaign No JOIN or ad hoc queries: every new question is a table or a CQRS view; consistency level chosen on every operation (a silent bug if chosen badly); operating compactions, repairs and snapshots 03-02, 04-04
ADR-003 Orchestrated sagas from orders (order_saga.py, sagas table) to confirm an order Pure choreography (each service reacts to events: more decoupled, but the flow is implicit and hard to follow and to compensate in order); Step Functions (latency and cost per transition on a critical path of 1.2 M/month; lock-in) orders knows about inventory and payments (coupling accepted); the orchestrator is a stateful component that must be recovered on start-up; no isolation between sagas (countermeasures from 03-05: reservations with expiry, pending states) 03-05, 08-04
ADR-004 Kafka as the single event bus (orders.events keyed by order, delivery.*, inventory.alerts, audit.events), with an outbox in the producers and idempotent consumers RabbitMQ for everything (no retention or replay, which are essential for rebuilding CQRS views and feeding the lake); HTTP calls between services to notify (temporal coupling, symptom 2); an ESB with transformations (the lesson of SOA) Eventual consistency as the norm; ordering only within a partition; event schemas as contracts to version; one critical cluster (MSK) on which everything depends; finite retention except in the lake 02-04, 02-05, 08-01
ADR-005 Kubernetes (EKS) with a service mesh, GitOps and canaries as the deployment platform for the services; managed services for data (RDS, MSK, ElastiCache, S3) Virtual machines with Ansible for everything (no autoscaling or reconciliation, manual deployments, symptom 3); a PaaS such as Cloud Run/App Runner per service (less control over networking, mesh and operators; it would fit a small version); self-managed Kubernetes (operating the control plane for no benefit) Operational complexity (namespaces, RBAC, mesh, operators) taken on by a platform team; 1-2 ms per sidecar hop; selective lock-in on the managed services; a permanent baseline cost (≈ €3,900/month with reservations) 07-05, 08-03
ADR-006 Serverless (Lambda with SAM) for sporadic event tasks: thumbnails, invoices, gateway webhooks, scheduled tasks; Step Functions only for long, rare workflows (producer offboarding) Consumers on Kubernetes for everything (permanent cost for tasks that run for seconds a day; probes and on-call for no reason); serverless for orders or the WebSocket (latency, state, connections); Step Functions for the order saga (ADR-003) High lock-in in the adapters (limited by separating handler and logic); automatic retries that force strict idempotency; cold starts; observability that has to be joined to the cluster's; revisit the cost if the frequency multiplies by ten 08-04

Every ADR lives in km0/docs/adr/ADR-00N-*.md, in the full format from 08-01. Note that three of the six decisions (002, 003, 006) explicitly reject a "more modern" or "more managed" option: the architecture is not the sum of the best technologies but of the decisions that answer a symptom at a known cost.

  1. The five symptoms of 01-06, revisited

Symptom (01-06) What it demanded How it was resolved How it is verified
1. Campaign spikes take everything down (Grape Harvest ×15; 92% catalogue reads) Scale the catalogue separately, serve reads from a cache and photos from elsewhere, isolate failures catalog as its own service with Redis cache-aside (04-05), photos in S3 with thumbnails by Lambda (04-03, 08-04) and a CDN with immutable (08-04); HPA and spot nodes during campaigns (07-05, 08-03); rate limiting at Kong (06-05); load shedding (07-04) k6 tests/load/harvest.js at ×15 in staging before every campaign (07-06): catalogue p99 < 100 ms, CDN hit ratio > 95%, orders without degradation
2. A payments failure takes everything down (gateway at 30 s; connections exhausted) payments isolated with its own resources, slowness contained, confirmation without a global transaction payments with an ACL in front of the gateway (08-01); timeout, circuit breaker and bulkhead on orderspayments (07-04); saga with expiring stock reservation and a "payment pending" state (03-05); its own database (ADR-001/002) Chaos experiment: Toxiproxy adds 30 s of latency to the gateway (07-06); the circuit opens in < 10 s, catalogue and tracking carry on, orders stay pending and complete when the circuit closes
3. Teams step on each other when deploying (4 teams, Tuesday-and-Thursday deployment) Independent deployment units per domain, with no downtime One Deployment per service with rolling updates and automatic canaries against the SLO (07-05); contract tests that protect consumers (07-06); GitOps: deploying is merging; teams owning services and on-call per service (08-01) Deployment frequency per service (DORA metric): delivery several times a day, payments once per sprint, with no coordination; zero coordinated deployments in the last quarter
4. Real-time courier telemetry (140 couriers, 2.4 M positions/day in the orders table; customers asking) A channel for high-frequency streams, bidirectional communication, time-series storage MQTT with QoS 1, ACL and sessions (08-02); bridge to delivery.positions with a key (02-04); Cassandra for positions (04-04); Flink for the dashboard (05-04); WebSockets with fan-out via Redis to Anna and to the operators (08-02); aggregator on the van (08-04) End-to-end latency (van → browser) p95 < 3 s; no position writes to the orders database; km0_ws_drops_total ≈ 0 outside incidents
5. Analytics chokes production (3 h of nightly queries against production) Separate the analytical load with a copy fed by events analytics consumes orders.events and delivery.* into the lake (04-02); Spark daily_sales.py on ephemeral EMR with spot (05-03, 08-03); Flink for what cannot wait for the batch (05-04); Airflow km0_daily_sales (05-05); a separate km0_analytics Zero queries from analytics against km0_inventory or km0_orders (network policy and credentials prevent them: 06-04, 07-05); the DAG finishes before 06:00 with an SLA in Airflow; producers have forecasts they previously did not dare to ask for

  1. Assessment against the course's criteria

A good course leaves criteria, not just techniques. These are the five with which Kilometre Zero assesses itself, and with which the student can assess any other platform.

The eight fallacies (01-04)

Fallacy Where the platform assumes it is false
The network is reliable Timeouts on every call (07-04); retries with idempotency (02-05); QoS 1 and a persistent queue on the van (08-02, 08-04); replication and quorums (03-04)
Latency is zero At most two synchronous calls on the critical path (08-01); caching at three tiers (04-05); CDN and Worker (08-04); measured latency SLO (07-01)
Bandwidth is infinite Binary gRPC (02-03); MQTT with a 2-byte header (08-02); coalescing towards slow clients (08-02); leg aggregation on the van and a CDN for egress (08-04)
The network is secure TLS everywhere (06-02); mTLS with the mesh (06-04); JWT verified at the edge, the gateway and the service (06-01, 06-05, 08-04); ACL on the broker; security groups and private subnets (08-03)
Topology doesn't change Service discovery via Kubernetes and the mesh (07-05); consistent hashing (04-01); reconnection with backoff in clients (08-02); RDS failover via DNS (08-03)
There is one administrator Teams owning services, an internal platform, Kubernetes RBAC and IAM per service account (08-01, 07-05, 08-03); contracts and ADRs as agreements
Transport cost is zero Egress, cross-AZ traffic and NAT in the budget; VPC endpoints; client.rack; FinOps (08-03)
The network is homogeneous Protobuf contracts and versioned event schemas (02-03, 02-05); the last mile designed for mobile networks (08-02)

Consistency model per piece of data (03-01, 03-02)

Data Model Mechanism Who reads it that way
Available units of a SKU per market Strong (linearizable per row) SELECT FOR UPDATE on the RDS primary; synchronous replica inventory.ReserveStock
Charge status Strong in payments; idempotent towards the gateway Local transaction + Idempotency-Key The saga
Anna's order (by id) Quorum (read-your-own-writes guaranteed by QUORUM+QUORUM) Cassandra RF=3 orders on confirmation; Anna when checking her order
A customer's order list Eventual (ONE) Cassandra The "my orders" screen
Availability on the catalogue listing Eventual (seconds) products_view via events + Redis + CDN 60 s Anna browsing
Producer dashboard Eventual (seconds) CQRS projection Martha Hill
Van position Eventual, "last wins" by seq MQTT retained, Redis last:, deduplication by sequence Anna, operators
Physical stock at the market stall Multi-leader with local authority; CRDTs for counters Reconciliation on sync The Lleida terminal
Sales reports Eventual (hours) Lake + nightly Spark Producers, management
Audit Strong on write, immutable Outbox → audit.events → S3 with object lock Compliance

The lesson of this table is that "strong" appears twice and "eventual" six times: strong consistency is paid for and reserved for what truly needs it.

SLOs (07-01) and RPO/RTO (07-03)

Service / data SLO RPO RTO
orders (create) 99.9% availability; p99 < 500 ms (excluding the gateway) km0_orders: 0 with QUORUM for a single node; minutes on loss of a region (snapshots) Node: 0 (no failover); region: 1-2 h (pilot light runbook)
inventory 99.95%; p99 < 100 ms on ReserveStock 0 (synchronous Multi-AZ replica); minutes across regions ~60 s (RDS failover); region: 30-60 min
catalog 99.9%; p99 < 200 ms (100 ms with the CDN) N/A (rebuildable from events and S3) Minutes (re-read orders.events)
delivery real time 99%; end-to-end latency p95 < 3 s Positions: losses are tolerated Minutes
Kafka 99.95% 0 with acks=all, min.insync.replicas=2 Seconds (leader election)
analytics Daily report before 06:00 (Airflow SLA) Hours Hours (re-run the DAG)
Photos and invoices 99.99% (S3) ~0 (versioning + CRR) Minutes (switch region at the CDN)

Security controls (Module 6): who can do what

Actor Identity Can Cannot Control
Anna (customer) Keycloak JWT, roles: [customer], client web-km0 View the catalogue, create her own orders, view her orders and invoices, track her deliveries over WebSocket, chat with her producers View Mark's orders; subscribe to a whole market; publish products RBAC in services (06-01); authorization per subscription (08-02); rate limit per consumer (06-05)
Martha Hill (producer) JWT, roles: [producer], claim producer_id: montblanc-dairy Publish and edit her products, upload photos via presigned URL, view her orders dashboard, receive stock alerts over SSE, reply in the chat Touch La Vega Farm's products; see customers' personal data beyond name and delivery ABAC "her products" in catalog (06-01); presigned URL with prefix (04-03); projection filtered by producer (08-01)
Jordan Hall (operator) JWT, roles: [operator] Delivery dashboard for every market, reassign vans, send commands over MQTT (via delivery), view the delivery audit trail Modify prices; access payment data RBAC; delivery publishes on commands with the bridge identity (08-02); audit of every command
van-3 (device) Its own MQTT user, credential in Vault, revocable Publish on km0/delivery/van-3/{position,status,leg}, read its commands Publish as van-7; read anything else Mosquitto ACL (08-02); validation at the bridge; local encryption (08-04)
orders (service) mTLS certificate issued by the mesh (SPIFFE), client orders-service in Keycloak, service account with an IAM role ReserveStock, GetStock on inventory; Charge on payments; produce on orders.events; read its own secrets Read km0_inventory directly; produce on delivery.*; access S3 ServiceInterceptor per method (06-04); mesh and network authorization policies (07-05); IRSA (08-03); dynamic Vault credentials
Lambda invoices The function's IAM role Consume orders.events (own group), write to km0-invoices, PutItem on km0-idempotency Read km0-photos; write to Kafka SAM policies (08-04)
CDN Worker No secret credential Verify signatures with the public key, cache public responses Access databases; see the private key "Nothing secret at the edge" design (08-04)
Platform team IAM with MFA, Kubernetes RBAC per namespace Operate the infrastructure, apply Terraform from the pipeline Read decrypted payment data; apply Terraform from a laptop against production Least privilege, GitOps, CloudTrail audit (08-03)

  1. What is left out and next steps

No architecture is ever finished; this one has five open fronts, each with the course module that provides the tools to tackle it:

  1. Active-active multi-region. Today there is a pilot light (07-03, 08-03): the secondary region takes 1-2 hours to serve. Moving to active-active would require deciding the write model per piece of data: km0_orders in multi-DC Cassandra (LOCAL_QUORUM per region, 04-04) is natural; stock (inventory) is not, and would force partitioning authority by market (Girona and Lleida write in eu-west-1, Valencia in eu-central-1) with the multi-leader model of 03-04 for the markets split across regions. It is a months-long project and the pending ADR-007.
  2. Full event sourcing. Today events are notifications of state changes that are stored separately (and forever in the lake). Event sourcing would make the events the source of truth for orders: state is rebuilt by replaying order.created, payment.confirmed, ... You gain perfect auditability and the ability to rebuild any view; you pay with snapshots, versioning of historical events and queries that always go through a projection. CQRS (08-01) has already prepared the ground.
  3. Data mesh. analytics is today a central team that consumes everything. As the domains grow, each team would start publishing its data as a product (with a schema, quality, an SLA and an owner) and analytics would become a platform. It is Conway's law (08-01) applied to data.
  4. The internal platform as a product. The platform team from 08-01 exists; what is missing is treating it as a product: a portal where a team creates a new service with its Deployment, its dashboards, its alerts, its pipeline and its template ADR in minutes (exercise 1 puts this to the test).
  5. Cost as a design constraint. FinOps (08-03) measures; the next step is for cost to enter architecture decisions the way latency does: euros per order as an SLO, a quarterly review of managed versus self-run services, and automatic shutdown of anything not serving traffic.

  1. The project: a reduced Kilometre Zero in docker-compose

The course has shown code in every lesson; the project proposes building a reduced, working version on your laptop, with what fits in docker-compose and no cloud: catalog, orders, inventory, Kafka, PostgreSQL, Redis and Prometheus. payments is simulated, delivery and analytics are optional, and the rest of the platform (Keycloak, Kubernetes, Terraform, Lambda, CDN) is outside the minimum scope. What matters is not the size but that every milestone is verifiable with a command.

Final structure of the km0/ repository

km0/
├── README.md
├── docker-compose.yml              # PostgreSQL, Kafka (KRaft), Redis, Prometheus, Grafana, the services
├── docs/
│   └── adr/                        # ADR-001 … ADR-006
├── contracts/
│   ├── inventory.proto             # ReserveStock, GetStock, WatchChanges (02-03)
│   └── events/                     # JSON schemas for order.created, stock.updated, ... (02-05)
├── services/
│   ├── common/                     # metrics.py, logs.py, traces.py, resilience.py, jwt.py
│   ├── catalog/                    # api.py, cache.py (04-05), stock_projection.py (08-01)
│   ├── orders/                     # http_api.py (v1/v2), order_saga.py (03-05), outbox_relay.py, repository.py
│   ├── inventory/                  # api.py (08-01), grpc_server.py, _domain.py, _repository.py, _events.py
│   ├── payments/                   # simulated_gateway.py, consumer.py
│   ├── delivery/                   # van_mqtt.py, mqtt_kafka_bridge.py, ws_server.py, ws_kafka_consumer.py
│   └── analytics/                  # lake_consumer.py
├── sql/
│   ├── inventory/                  # skus, reservations, outbox, processed_messages
│   ├── catalog/                    # products, products_view.sql
│   └── orders/                     # sagas (in the reduced version, PostgreSQL instead of Cassandra)
├── simulations/                    # clocks, partitions, quorums (Modules 1 and 3)
├── dags/                           # km0_daily_sales.py (05-05)
├── serverless/                     # thumbnails/, invoices/, template.yaml, saga/ (08-04)
├── edge/
│   ├── mosquitto/                  # mosquitto.conf, acl, passwd (08-02)
│   ├── web/tracking.js             # WebSocket client (08-02)
│   ├── worker/catalog.js           # CDN Worker (08-04)
│   └── van/aggregator.py           # (08-04)
├── certs/                          # km0-ca and development certificates (06-04)
├── observability/
│   ├── prometheus.yml, alerts.yml  # (07-01)
│   └── grafana/dashboards/         # RED per service, saga, consumer lag
├── k8s/                            # orders-deployment.yaml, HPA, PDB, NetworkPolicy, overlays/ (07-05)
├── ansible/                        # inventory and Kafka/Cassandra playbooks (07-05)
├── infra/aws/                      # main.tf, variables.tf, outputs.tf, environments/ (08-03)
└── tests/
    ├── integration/                # Testcontainers: saga, idempotency, projection (07-06)
    ├── contract/                   # Pact, protobuf and event compatibility
    ├── load/harvest.js             # k6
    └── chaos/                      # Toxiproxy, Chaos Mesh manifests

docker-compose.yml for the reduced version

# km0/docker-compose.yml — reduced version for the final project
services:
  postgres:
    image: postgres:16
    environment: { POSTGRES_USER: km0, POSTGRES_PASSWORD: km0, POSTGRES_DB: km0 }
    volumes:
      - ./sql:/docker-entrypoint-initdb.d:ro      # creates km0_inventory, km0_catalog, km0_orders and their tables
    ports: ["5432:5432"]
    healthcheck: { test: ["CMD-SHELL", "pg_isready -U km0"], interval: 5s, retries: 10 }

  kafka:
    image: apache/kafka:3.7.0                      # KRaft: no ZooKeeper (03-03)
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"    # topics are created explicitly (02-04)
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    ports: ["9092:9092"]

  kafka-init:                                      # creates the topics and exits
    image: apache/kafka:3.7.0
    depends_on: [kafka]
    entrypoint: ["/bin/sh", "-c"]
    command: |
      "sleep 5 &&
       /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 --create --if-not-exists --topic orders.events --partitions 6 --replication-factor 1 &&
       /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 --create --if-not-exists --topic inventory.alerts --partitions 3 --replication-factor 1 &&
       /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 --create --if-not-exists --topic orders.events.dlq --partitions 1 --replication-factor 1"

  redis:
    image: redis:7
    ports: ["6379:6379"]

  inventory:
    build: ./services/inventory
    environment: { PG_DSN: "postgresql://km0:km0@postgres/km0_inventory", KAFKA_BOOTSTRAP: "kafka:9092" }
    depends_on: { postgres: { condition: service_healthy } }
    ports: ["50051:50051", "9101:9100"]           # gRPC and metrics

  orders:
    build: ./services/orders
    environment:
      PG_DSN: "postgresql://km0:km0@postgres/km0_orders"
      KAFKA_BOOTSTRAP: "kafka:9092"
      INVENTORY_GRPC: "inventory:50051"
      PAYMENTS_URL: "http://payments:8002"
    depends_on: [inventory, kafka-init]
    ports: ["8001:8000", "9102:9100"]

  payments:
    build: ./services/payments                     # simulated gateway: rejects 5% and takes 200-1500 ms
    ports: ["8002:8002"]

  catalog:
    build: ./services/catalog
    environment:
      PG_DSN: "postgresql://km0:km0@postgres/km0_catalog"
      KAFKA_BOOTSTRAP: "kafka:9092"
      REDIS_URL: "redis://redis:6379"
    depends_on: [redis, kafka-init]
    ports: ["8003:8000", "9103:9100"]

  prometheus:
    image: prom/prometheus:v2.53.0
    volumes: ["./observability/prometheus.yml:/etc/prometheus/prometheus.yml:ro"]
    ports: ["9090:9090"]

  grafana:
    image: grafana/grafana:11.1.0
    volumes: ["./observability/grafana:/etc/grafana/provisioning:ro"]
    ports: ["3000:3000"]

Verifiable milestones

Milestone What to build How to check it
M1. Start-up docker compose up brings everything up; the three services expose /health and /metrics curl localhost:8001/health{"status":"ok"}; curl localhost:9090/api/v1/targets shows all three up
M2. Catalogue with cache GET /products/aged-cheese with cache-aside in Redis (04-05) First request > 5 ms and X-Cache: MISS; second < 1 ms and HIT; redis-cli keys 'product:*' shows the key with a TTL
M3. Stock reservation over gRPC inventory.ReserveStock with SELECT FOR UPDATE, idempotent by reservation_id, publishing to outbox (02-03, 08-01) grpcurl -plaintext -d '{"reservation_id":"r1","order_id":"P-1","market":"girona","lines":[{"product":"aged-cheese","units":2}]}' localhost:50051 km0.inventory.v1.Inventory/ReserveStock twice → same response, stock deducted once; SELECT * FROM outbox with stock.reserved and stock.updated
M4. Outbox → Kafka Relay that reads outbox and produces to orders.events keyed by order, marking them as published (02-05) kafka-console-consumer --topic orders.events --from-beginning --property print.key=true shows the events with their envelope; killing the relay halfway and restarting does not duplicate (unique event_ids)
M5. Create order with saga POST /orders with Idempotency-Key; saga reserve → charge → confirm; compensation if payments rejects (03-05) curl -X POST -H 'Idempotency-Key: k1' ... twice → a single order; with the gateway forced to reject (PAYMENTS_MODE=reject), the order ends up rejected and the stock comes back (SELECT available FROM skus); SELECT * FROM sagas shows the transitions
M6. CQRS projection catalog consumes stock.updated, updates products_view and invalidates Redis (08-01) After M5, GET /products/aged-cheese shows the updated available in < 2 s; SELECT * FROM processed_messages grows; re-sending an event by hand with the same event_id changes nothing
M7. Resilience Timeouts, retries with jitter and a circuit breaker on orderspayments (07-04) Stop payments (docker compose stop payments): requests fail fast (< 1 s, not 30 s) once the circuit opens; km0_circuit_state{dependency="payments"} = 2 in Prometheus; when payments comes back, it closes by itself
M8. Observability RED per service, saga latency, consumer lag; a Grafana dashboard; an SLO alert (07-01) Dashboard with rate(km0_orders_total[1m]) by status and p99 of POST /orders; the alert fires when 5% errors are provoked with PAYMENTS_MODE=errors
M9. Load k6 with 50 virtual users creating orders and reading the catalogue for 2 minutes (07-06) Catalogue p99 < 50 ms; no duplicate orders (SELECT count(*), count(DISTINCT idempotency_key) equal); final stock = initial − sold
M10. Integration test Testcontainers bringing up PostgreSQL and Kafka; a test of the saga with a rejecting payments and of consumer idempotency (07-06) pytest tests/integration green in under 2 minutes
Optional A. Real time Mosquitto + mqtt_kafka_bridge.py + ws_server.py + tracking.js (08-02) Publish with mosquitto_pub -t km0/delivery/van-3/position -q 1 -m '{...}' and see it in the browser; publish a lower seq and check that it is discarded
Optional B. Analytics A consumer that writes events to Parquet files by date; a local Spark script that computes sales per producer (05-03) spark-submit daily_sales.py --date 2026-09-15 produces the table with Montblanc Dairy at the top

Every milestone should end with a short ADR if a decision is taken (for example, "PostgreSQL instead of Cassandra for orders in the reduced version: context, consequences") and with the tests that verify it in tests/. The result is a repository you can show and talk about in an interview with the same precision with which this course has talked about Kilometre Zero.

  1. Study guide

The course has been a map; these are the territories to explore next. Works and papers are cited by title and author, without links: all of them are easy to find by name.

Reference books

  • Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017). The book that most resembles Modules 2, 3 and 4 of this course, in far greater depth: replication, partitioning, transactions, consensus, batch and stream processing. It is the natural next read.
  • Andrew S. Tanenbaum and Maarten van Steen, Distributed Systems (3rd ed., available free of charge from the authors). The classic academic text: models, communication, synchronisation, consistency and fault tolerance with formal rigour. It complements Modules 1 and 3.
  • Sam Newman, Building Microservices (2nd ed., O'Reilly, 2021) and Monolith to Microservices (2019). Everything in 08-01 in detail: boundaries, migration, testing, organisation.
  • Betsy Beyer et al. (eds.), Site Reliability Engineering: How Google Runs Production Systems and The Site Reliability Workbook. The origin of the SLOs, error budgets, postmortems and on-call of Module 7. Available free online from Google.
  • Eric Evans, Domain-Driven Design (2003), and Vaughn Vernon, Implementing Domain-Driven Design (2013), to go deeper into the bounded contexts of 08-01.
  • Gregor Hohpe and Bobby Woolf, Enterprise Integration Patterns (2003): the catalogue of messaging patterns from which 02-04 and 02-05 took their names.
  • Neal Ford, Mark Richards et al., Software Architecture: The Hard Parts (2021): trade-offs of decomposition, distributed data and sagas, in the same ADR spirit.

Foundational papers (all findable by title)

  • Leslie Lamport, "Time, Clocks, and the Ordering of Events in a Distributed System" (1978): the basis of 01-05. And "The Part-Time Parliament" (1998) and "Paxos Made Simple" (2001) for consensus.
  • Diego Ongaro and John Ousterhout, "In Search of an Understandable Consensus Algorithm" (Raft, 2014): what etcd and Kafka KRaft implement (03-03).
  • Giuseppe DeCandia et al., "Dynamo: Amazon's Highly Available Key-value Store" (2007): consistent hashing, quorums, vector clocks and the choice of availability that Cassandra inherits (04-01, 04-04).
  • Jeffrey Dean and Sanjay Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters" (2004), and Sanjay Ghemawat et al., "The Google File System" (2003): 05-02 and 04-02.
  • James C. Corbett et al., "Spanner: Google's Globally-Distributed Database" (2012): strong consistency at global scale with bounded clocks (TrueTime), the counterpoint to Dynamo.
  • Jay Kreps, Neha Narkhede and Jun Rao, "Kafka: a Distributed Messaging System for Log Processing" (2011), and Jay Kreps's essay "The Log: What every software engineer should know about real-time data's unifying abstraction" (2013): the idea of the log as the backbone (ADR-004).
  • Hector Garcia-Molina and Kenneth Salem, "Sagas" (1987): the original paper behind 03-05.
  • Seth Gilbert and Nancy Lynch, "Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services" (2002), and Daniel Abadi, "Consistency Tradeoffs in Modern Distributed Database System Design" (PACELC, 2012): 03-02.
  • Marc Shapiro et al., "Conflict-free Replicated Data Types" (2011): the CRDTs of 03-01 and 08-04.
  • Peter Deutsch and James Gosling, the "Fallacies of Distributed Computing", and Peter Bailis et al., "Highly Available Transactions: Virtues and Limitations" (2013), to continue with the fallacies and isolation models.
  • Tyler Akidau et al., "The Dataflow Model" (2015): event time, watermarks and windows from 05-04.

Practice

  • Jepsen (Kyle Kingsbury): public analyses of how real databases fail under partitions; the best school of scepticism about advertised guarantees.
  • Michael Nygard's Release It! (2nd ed., 2018), for the stability patterns of 07-04 told through real incidents.
  • Build the project from section 9 and then break it with the experiments from 07-06.

Common Mistakes and Tips

  • Presenting the architecture as a diagram without decisions. The diagram in section 1 is worth little without the ADRs in section 5; what you learn from an architecture is its rejected alternatives and its accepted consequences.
  • Assessing by technologies rather than by criteria. "It uses Kafka and Kubernetes" says nothing about whether it is well designed. The tables in section 7 (fallacies, consistency per piece of data, SLOs, RPO/RTO, who can do what) do.
  • Starting the project with the infrastructure. Kubernetes, mesh and Terraform with no service to deploy is the mistake of "choosing technologies before problems" from 01-06. The project starts at M1 with three services and docker-compose.
  • Believing the platform is finished. Section 8 lists five open fronts; an architecture without a backlog is one nobody is looking at.
  • Tip: in any new system, start by writing down the symptoms (01-06), the consistency-per-data table (03-01) and the SLOs (07-01) before drawing boxes. They are the three pages that produce the most correct decisions.
  • Tip: keep this course as a reference by lesson: when in a year's time you have to decide between orchestration and choreography, or how much TTL to put on a CDN, the corresponding lesson has the table.

Exercises

Exercise 1: bringing in the "reviews" domain

In 08-01 it was decided that reviews are a bounded context of their own with event-based communication. Now design it end to end applying the whole course: (a) the contract of events it consumes and produces (with the envelope from 02-04) and the data model with its consistency model (03-01) and its storage (Module 4); (b) the public API at Kong with authentication and authorization (who may review what; 06-01, 06-05) and how the average reaches the listing cached by the CDN (08-04); (c) SLO, metrics and alerts (07-01), and what traces and audit records a review leaves behind; (d) deployment (07-05), RPO/RTO (07-03) and what would have to be added to main.tf (08-03); (e) ADR-007 in summary format (decision, rejected alternative, consequence).

Exercise 2: an end-to-end incident

Saturday 19 September, 18:42, Artisan Cheese Week. The alert "orders SLO: error budget burn ×14 in 5 min" fires. Available data:

  • Prometheus: rate(km0_orders_total{status="rejected_stock"}[5m]) goes from 0.3/s to 9/s at 18:40; the p99 latency of POST /orders drops from 420 ms to 95 ms; km0_grpc_client_duration_seconds{method="ReserveStock"} p99 = 8 ms; km0_circuit_state{dependency="inventory"} = 0; lag of the catalog-projection group = 0.
  • inventory logs (Loki): from 18:39:50, hundreds of lines {"level":"warning","msg":"InsufficientStock","product":"aged-cheese","market":"girona","available":0,"requested":2,"request_id":...}.
  • catalog logs: {"level":"info","msg":"view updated","product":"aged-cheese","market":"girona","applied":true} at 18:39:48, and none after that for that product.
  • Traces: the rejected orders have spans kong → orders → inventory.ReserveStock (InsufficientStock) of 90 ms; none reaches payments.
  • products_view in catalog: aged-cheese / girona: available = true, approx_units = 0, stock_version = 1758300000123.
  • Audit: at 18:39:45, {"sub":"mhill","action":"stock.adjust","resource":"aged-cheese/girona","from":140,"to":0,"source":"producer-dashboard"}.

(a) Reconstruct the complete causal chain. (b) Is it a platform incident, a product incident or an operations incident? Which part of the behaviour is correct and which is a defect? (c) Locate the exact defect in the code from 08-01 (stock_projection.py or products_view.sql) and fix it. (d) Write the three postmortem actions (07-03) and say which signal would have detected the problem before the SLO did.

Exercise 3: trimming the architecture down for three people

A three-person startup wants to launch a marketplace like Kilometre Zero in a single market, with 20 producers, 300 orders a day and 6 couriers. They have three months and cannot operate Kubernetes or Kafka. Using the "when not to use microservices" table from 08-01 and the selective lock-in of 08-03, design the minimum architecture: (a) what stays as a modular monolith and with what internal structure; (b) which three Kilometre Zero decisions you would keep intact because they are cheap and prevent irreversible mistakes; (c) how you would handle real-time tracking and photos without Kafka, MQTT or Lambda; (d) which measurable signals would indicate that the time has come to extract the first service, and which one it would be.

Solutions

Exercise 1.

(a) It consumes order.delivered (published by delivery on orders.events with the standard envelope; data: {order_id, customer, lines: [{product, producer}], delivered_ms}) to know what can be reviewed. It produces on a new topic reviews.events (3 partitions, key = product, so that a product's reviews reach the projection in order): review.created {review_id, customer, order_id, product, producer, points, comment, timestamp_ms} and review.replied {review_id, producer, reply, timestamp_ms}. Model: tables reviewable (customer, order_id, product, delivered_ms, reviewed bool) and reviews (id, customer, product, producer, points, comment, reply, created_at) in its own PostgreSQL database km0_reviews (a small RDS, without Multi-AZ at first): low volume (one per delivered line), relational queries (by product, by producer, awaiting reply), and the rule "one review per line" requires a UNIQUE (customer, order_id, product) constraint with local strong consistency. The average on the listing is eventual (a projection in catalog), and the producer dashboard reads its own database with strong consistency.

(b) Routes at Kong: POST /api/v1/reviews (role customer), GET /api/v1/products/{slug}/reviews (public, cacheable for 60 s), POST /api/v1/reviews/{id}/reply (role producer), GET /api/v1/producers/me/reviews?pending=true (role producer). Authorization in the service: on creation, claims.sub must have an unreviewed row in reviewable for that order and product (ABAC on its own data, without calling orders); on reply, claims.producer_id must match reviews.producer. Specific rate limiting (5 reviews/minute per consumer) against abuse. The average reaches the listing because catalog consumes review.created and maintains points_sum and review_count in products_view (idempotent by event_id); the listing stays public, max-age=60 per market, and the average changes with up to a minute's delay at the CDN, which is acceptable. The three latest comments are served on the public reviews route (cacheable, with no personal data beyond the first name, which the customer consented to).

(c) SLO: 99.5% availability and p99 < 300 ms for creation (it is not on the purchase path: a looser SLO than orders). Metrics: km0_reviews_total{result}, latency per route, lag of the order.delivered consumer and of the catalog consumer on reviews.events. Alerts: error budget burn, lag > 5 min (a delivery that cannot be reviewed), and an anomalous rate of unauthorized (abuse). Trace: kong → reviews POST → postgres, linked to the catalog consumer by the event_id. Audit: reviews.create and reviews.reply on audit.events (producers' replies are public content, and moderation needs to know who wrote what).

(d) k8s/reviews-deployment.yaml copying the orders template (2 replicas, probes, resources, topologySpreadConstraints, HPA by CPU, a NetworkPolicy that only allows ingress from Kong and egress to its RDS and to Kafka, serviceAccountName with IRSA and no S3 permissions). RPO: minutes (RDS backups and PITR; losing reviews is tolerable but annoying); RTO: 1 hour (it does not block sales). In main.tf: aws_db_instance.reviews (db.t4g.small, no multi_az), its security group (5432 only from EKS), an IRSA module with no S3 policies, and the reviews.events topic managed with the kafka provider or created by the pipeline.

(e) ADR-007: "reviews as its own bounded context with PostgreSQL and communication exclusively through events". Rejected alternative: inside catalog (it would couple moderation to the catalogue's deployments during campaigns and mix two languages: "published product" and "review"). Accepted consequence: the listing shows the average with up to a minute's delay; one more component with a database, a dashboard and on-call (owned by the catalogue and producers team); if delivery stops publishing order.delivered, nobody can review until the lag recovers (alert).

Exercise 2.

(a) Causal chain: at 18:39:45, Martha Hill adjusts the stock of aged-cheese in Girona from 140 to 0 from her dashboard (probably a data-entry mistake, or a genuine withdrawal of the product). inventory applies the adjustment and publishes stock.updated {available: 0}. At 18:39:48, catalog receives the event and the projection applies it (applied: true) → approx_units = 0. But available stays true. From 18:39:50, customers who see the listing (with available = true) add aged-cheese to their basket and confirm; orders starts the saga, inventory answers InsufficientStock in 8 ms, the saga compensates and the order ends up rejected_stock. The p99 latency drops (rejected orders never go through the gateway), the rejection rate rises ×30 and the error budget is consumed because rejected_stock counts as a failure of the orders SLO.

(b) It is a mixed incident. The saga, the circuit and the compensation work correctly: no order was charged without stock, no reservation was left hanging. Martha's adjustment is a legitimate business action (and the audit records it precisely). The defect lies in the platform: the catalogue listing keeps saying "available" with 0 units, steering hundreds of customers into a certain rejection. It is a defect in the CQRS projection, not in eventual consistency: the delay was 3 seconds, which is acceptable; the problem is that the projected data is incorrect permanently.

(c) In stock_projection.py, apply computes avail = d["available"] >= AVAILABLE_THRESHOLD with AVAILABLE_THRESHOLD = 1: with available = 0, avail = False, and the UPDATE should have set available = FALSE... unless the stock_version < %(v)s condition failed. But the log says applied: true, so the UPDATE did affect the row. So the defect is not there; look at products_view.sql: available BOOLEAN NOT NULL DEFAULT FALSE and approx_units are updated together in the same UPDATE. The row shows approx_units = 0 and available = true, which is only possible if another write path set available = true afterwards: the product publication path used by the producer (catalog creates/updates the row when the producer edits the listing and, according to 08-01, "the row is created by catalog when the producer publishes the product"). Martha edited the listing (perhaps the same adjustment from the dashboard touches both product and stock), and the publication UPSERT overwrites available with its default value or with the value the listing had in the form, without respecting stock_version. The defect is that two paths write the same column with different rules: the fix is for listing publication to never write available or approx_units (columns owned exclusively by the projection), that is, INSERT ... ON CONFLICT (slug, market) DO UPDATE SET name = EXCLUDED.name, price_cents = EXCLUDED.price_cents, producer = EXCLUDED.producer without touching the stock columns; and, as a safety net, a CHECK (NOT available OR approx_units >= 1) on the table that would make the incoherent write fail rather than let it through. It is the "data owner" principle of 08-01 applied inside a service, column by column.

(d) Blameless postmortem (07-03): 1. Fix the publication UPSERT and add the CHECK; an integration test (07-06) that publishes the listing after a stock.updated to 0 and verifies available = false. 2. Add a confirmation in the producer dashboard for stock adjustments that change more than 50% ("Set aged-cheese in Girona to 0 units? There were 140"), because the audit shows an improbable jump. 3. Separate stock rejections (a correct business outcome) from platform errors in the orders SLO, or create a specific "rejected_stock rate per product" alert with a low threshold, so that the symptom is attributed properly. The signal that would have caught it earlier: a coherence alert on the projection, count(products_view where available and approx_units = 0) > 0, evaluated every minute (a cheap query), or the Flink stock_alerts metric from 05-04 for aged-cheese in Girona, which fired at 18:39:50 towards inventory.alerts and which Martha received over SSE (08-02) but which nobody correlated with the rejections.

Exercise 3.

(a) A modular monolith in Python (FastAPI) with one managed PostgreSQL database (RDS or Cloud SQL with automatic backups and PITR, without Multi-AZ at first), deployed on a container PaaS (Cloud Run, App Runner, Fly.io, Render: 2 instances, autoscaling included, managed TLS). Internal structure: the six packages from 01-06 (catalog, orders, inventory, payments, delivery, analytics), each with its api.py as the only public module (08-01), PostgreSQL schemas separated per package (inventory.skus, orders.orders) and a lint rule (import-linter) that forbids importing another package's private modules and SQL queries across schemas. The order confirmation transaction is local (a single database): reservation, order and charge record in one transaction, with the gateway call outside it (charge first with an Idempotency-Key, then confirm in the transaction; if the confirmation fails, refund). A small managed Redis for the catalogue cache and the rate limiter. A single repository, one pipeline, docker-compose for development.

(b) Three cheap decisions that prevent irreversible mistakes: 1. The event envelope and an events table (outbox) from day one, even if the "bus" is the database itself and the consumer a process inside the monolith: when Kafka arrives, the producers already publish and the consumers are already idempotent; and the analytical lake can start by exporting that table to Parquet every night. 2. Idempotency in the write APIs (Idempotency-Key on create order, deterministic keys towards the gateway): it is what in 02-05 and 08-04 prevents duplicate charges and orders, it costs one table, and it cannot be added later without migrating clients. 3. Minimal observability with an SLO: JSON logs with request_id, RED metrics and a written SLO for create order; without this, there will be no data to decide when to extract anything (part d). And a free fourth one: ADRs from day one.

(c) Tracking: 6 couriers sending positions via HTTPS POST every 10 s to the monolith (60 requests/minute: nothing), stored in a delivery.positions table with 24 h retention (at this volume, PostgreSQL is more than enough), and customers using SSE or polling every 5 s against an endpoint that returns the order's last position (with Cache-Control: max-age=4 so that the PaaS or a CDN absorbs repeats). No MQTT or WebSockets: there is no volume to justify them, and SSE with automatic reconnection covers the experience. Photos: upload via presigned URL to S3/GCS (04-03, kept) and thumbnails generated in the monolith itself as a background task (a simple queue in PostgreSQL with SELECT ... FOR UPDATE SKIP LOCKED, processed by a worker in the same deployment), with a free or low-cost CDN in front of the bucket using immutable keys (08-04, kept because it is almost free and avoids egress).

(d) Signals for extracting the first service, measurable thanks to (b)-3: the p99 of create order degrades when catalogue traffic rises (contention on the same database or CPU: symptom 1); the team grows from 3 to 8-10 people and catalogue deployments break orders (symptom 3); read traffic exceeds what Redis plus a PostgreSQL read replica can absorb; or the gateway's latency starts to block connections (symptom 2). The first to go, as in 01-06, is catalog (reads only, no cross-cutting transactions, rollback with a route change) or, if the dominant symptom is the gateway, payments with its ACL. Never orders and inventory first: they are the ones that demand sagas, and sagas are only justified when the boundary already exists for another reason. The rule from 08-01 holds in reverse: three people with no symptoms is exactly the row of the table that says "modular monolith".

Conclusion

Kilometre Zero began as a Python monolith with one database and five symptoms, and ends as a platform that can be drawn in a diagram, walked through with an order, justified with six ADRs and assessed against the criteria the course itself has provided: the eight fallacies assumed false at specific points of the design, a consistency-per-data table where "strong" appears twice and "eventual" six times, SLOs and RPO/RTO per component, and a who-can-do-what table from Anna to the CDN Worker. Each of the five symptoms from 01-06 has its resolution and its way of being verified; each decision has its rejected alternative and its price; and what is left out is written down, with the module that provides the tools to tackle it. The proposed project reduces all of that to what fits on a laptop, with ten milestones that are checked with a command, and the study guide points to the books and papers where every idea in this course has its origin and its depth.

What the student can now do is not "use Kafka" or "deploy on Kubernetes", although that too. It is something more durable: faced with a system that has to be spread across several machines, they know to write down the symptoms first and not the technologies; to cut boundaries where the language changes meaning and not where there is a table; to choose for each piece of data the consistency model it needs and pay only for that; to give every call a timeout, every message an identifier and every consumer idempotency, because the network is going to fail; to replace the global transaction with a saga that compensates; to put data where its workload asks for it, from PostgreSQL to Cassandra, from Redis to S3, from the CDN to a van's SQLite queue; to separate what is computed in batches from what cannot wait; to authenticate and authorize at every boundary without trusting any; to measure with SLOs, observe with traces, recover with tested backups and rehearse failures before they happen; to describe infrastructure as code and cost as a metric; and to leave every decision written down with its context and its consequences, so that whoever comes next can change it knowingly.

That is what designing distributed architectures means: not piling up pieces, but knowing for each one what is gained, what is lost and why it was chosen. Kilometre Zero has been the pretext; the judgement is what the student takes away. Thank you for following the course this far, and good luck with the next system that has to be spread across several machines: you now know where to start.

Distributed Architectures Course

Module 1: Introduction to Distributed Systems

Module 2: Communication in Distributed Systems

Module 3: Consistency and Replication

Module 4: Distributed Storage

Module 5: Distributed Computing

Module 6: Security in Distributed Systems

Module 7: Monitoring and Maintenance

Module 8: Case Studies and Applications

© Copyright 2026. All rights reserved