In 01-06 we opened up a Python monolith with a PostgreSQL database, a Tuesday-and-Thursday deployment and five symptoms pushing it to change, and we drew a target architecture with a table that pointed each component at a future lesson. Thirty-eight lessons later, every one of those boxes has been built: gRPC and Kafka, sagas and replicas, Cassandra and Redis, Spark and Flink, Keycloak and Vault, Prometheus and Kubernetes, MQTT and WebSockets, Terraform and Lambda, the CDN and the van's SQLite queue. This last lesson adds no new piece: it assembles the ones we have. First the complete architecture in a single diagram and the table from 01-06 closed off; then the journey of one of Anna's orders from her browser to the invoice in S3, with everything it leaves in its wake in traces, metrics and audit records; then the six architecture decisions as ADRs, with their rejected alternatives and their consequences; the five symptoms of 01-06 with their resolution; the assessment of the platform against the course's own criteria (fallacies, consistency per piece of data, SLOs, RPO/RTO, security controls); what is left out; the project the student can build on their laptop with verifiable milestones; and a guide for going deeper. The exercises are integration exercises: bringing in a new domain, analysing an incident end to end and trimming the architecture down for three people. The conclusion closes the course.
Contents
- The final architecture of Kilometre Zero
- The table from 01-06, closed
- The journey of one of Anna's orders end to end
- What the order leaves in its wake: traces, metrics and audit
- The architecture decisions as ADRs
- The five symptoms of 01-06, revisited
- Assessment against the course's criteria
- What is left out and next steps
- The project: a reduced Kilometre Zero in
docker-compose - Study guide
- Common Mistakes and Tips
- Exercises
- Conclusion
- The final architecture of Kilometre Zero
flowchart TB
subgraph Clients
ANNA["Anna's browser / phone"]
CAPP[Courier app]
PDASH[Producer dashboard]
VAN[van-3<br/>SQLite aggregator]
end
subgraph Edge[Network edge]
CDN[CDN + Worker<br/>cache per market, JWT]
end
subgraph Region[Region eu-west-1, 3 AZs]
ALB[ALB] --> KONG[Kong<br/>jwt, rate-limiting, X-Request-Id]
KONG --> BFF[Courier app BFF]
subgraph EKS[EKS + service mesh mTLS]
CAT[catalog]
ORD[orders<br/>saga, outbox]
INV[inventory<br/>gRPC]
PAY[payments<br/>gateway ACL]
DEL[delivery<br/>ws_server, MQTT bridge]
ANL[analytics]
MQ[Mosquitto]
OBS[Prometheus · Loki · Tempo · Grafana]
KC[Keycloak realm km0]
VLT[Vault / Secrets Manager]
end
KONG --> CAT & ORD & DEL
BFF --> DEL & ORD & CAT
ORD -- gRPC mTLS --> INV
ORD -- gRPC mTLS --> PAY
PAY --> GW[(Payment gateway)]
subgraph Data
RDS[(RDS PostgreSQL Multi-AZ<br/>km0_inventory)]
CAS[(Cassandra 3 nodes<br/>km0_orders)]
RED[(ElastiCache Redis<br/>cache, rate limiter, pub/sub)]
S3[(S3 km0-photos, invoices,<br/>backups, audit)]
LAKE[(S3 lake /km0/events)]
ANDB[(PostgreSQL km0_analytics)]
end
INV --> RDS
ORD --> CAS
CAT --> RED
CAT --> S3
subgraph Messaging
MSK[(MSK Kafka<br/>orders.events, inventory.alerts,<br/>delivery.positions, delivery.dashboard,<br/>audit.events)]
end
ORD -. outbox .-> MSK
INV -. outbox .-> MSK
MSK -.-> CAT & DEL & ANL
MQ --> DEL --> MSK
subgraph Compute
FLK[Flink stock_alerts, dashboard]
SPK[Spark daily_sales, EMR spot]
AIR[Airflow km0_daily_sales]
end
MSK --> FLK --> MSK
MSK --> LAKE --> SPK --> ANDB
AIR --> SPK
subgraph Serverless
LTHUMB[Lambda thumbnails]
LINV[Lambda invoices]
end
S3 -- s3:ObjectCreated --> LTHUMB --> S3
MSK -- payment.confirmed --> LINV --> S3
end
subgraph DR[Region eu-central-1, pilot light]
RDSR[(RDS replica)]
S3R[(Replicated S3)]
MSKR[(Minimal MSK, MirrorMaker)]
end
RDS -.-> RDSR
S3 -.-> S3R
MSK -.-> MSKR
ANNA --> CDN --> ALB
PDASH --> CDN
CAPP --> ALB
VAN -- MQTT QoS 1 --> MQ
DEL -- WebSocket --> ANNA
KC -. OIDC .- ANNA & CAPP & PDASH
The diagram has one piece that until now was only promised: the courier app BFF, announced in 06-05 and placed in 08-01. It is a small service, owned by the delivery team, behind Kong, that composes the app's home screen (the day's route from delivery, the status of that route's orders from orders, product names and photos from catalog) into a single response, with parallel calls, short timeouts and graceful degradation (if catalog does not answer, the route is shown without photos). It has no database: it is API composition (08-01, section 5), justified because the app makes few calls, the data is small and it needs freshness.
- The table from 01-06, closed
The table in 01-06 was a promise: each component, and the lesson where it would be built. This is the same table, closed, with what was finally built and the file or name that materialises it in km0/.
| Component | Lesson | What was built |
|---|---|---|
| Protocols: TCP/UDP/HTTP; MQTT and WebSockets introduced | 02-01 | TCP client with timeout; header measurement |
| Synchronous RPC between services | 02-02 | Stubs, marshalling, failure semantics |
| gRPC with contracts | 02-03 | contracts/inventory.proto (ReserveStock, GetStock, WatchChanges) |
| Event bus and queues | 02-04 | Kafka orders.events (6 partitions, key = order), envelope event_id/type/version/timestamp_ms/source/data |
| Idempotency, outbox, DLQ, competing consumers | 02-05 | outbox table, processed_messages, .retry topics and DLQ |
| Consistency models; CRDTs | 03-01 | Consistency-per-data table; G-Counter |
CAP and PACELC: inventory (C) and orders (A) |
03-02 | Decision per domain |
| Consensus: Raft, etcd, leader election | 03-03 | etcd for Patroni; Kafka controller |
| Replication: primary/replica, multi-leader, quorums | 03-04 | inv-bcn/inv-vlc; Patroni |
| Sagas | 03-05 | order_saga.py, sagas table, orchestration and choreography |
| Partitioning and consistent hashing | 04-01 | Ring, Kafka and Cassandra partitions |
| Distributed file systems | 04-02 | Lake /km0/events/... (HDFS, later S3) |
| Object storage | 04-03 | MinIO/S3 km0-photos, km0-invoices, km0-backups, km0-audit; photos.py; presigned URLs |
| Distributed databases | 04-04 | Cassandra km0_orders (orders_by_customer, orders_by_id), repository with consistency levels |
| Caches | 04-05 | Redis cache-aside for catalog, invalidation on stock.updated, Redis Cluster |
| Computing models | 05-01 | BSP, dataflow, actors |
| MapReduce and Hadoop | 05-02 | Sales per market over the lake |
| Spark | 05-03 | daily_sales.py |
| Streams | 05-04 | Flink stock_alerts.py (inventory.alerts), dashboard delivery.positions → delivery.dashboard |
| Pipelines | 05-05 | Airflow DAG km0_daily_sales |
| AuthN/AuthZ | 06-01 | JWT (sub, roles, jti), RBAC/ABAC in services |
| Encryption | 06-02 | TLS, encryption at rest with our own keys, field-level encryption for payments and positions |
| Identities | 06-03 | Keycloak realm km0, clients web-km0, courier-app, orders-service; OIDC |
| mTLS and secrets | 06-04 | km0-ca, AuthInterceptor/ServiceInterceptor, Vault (AppRole, dynamic credentials) |
| Gateway and auditing | 06-05 | Kong (/api/v1/catalog, /api/v1/orders, jwt, rate-limiting, X-Request-Id), audit.events, km0-audit |
| Metrics and SLOs | 07-01 | services/common/metrics.py, km0_orders_total, SLO 99.9% and p99 < 500 ms, error budget burn rate alerts |
| Logs and traces | 07-02 | services/common/{logs,traces}.py, Loki, Tempo, OpenTelemetry, X-Request-Id |
| Failures and recovery | 07-03 | Patroni + etcd, ISR, checkpoints, backups and PITR in km0-backups, RPO/RTO, pilot light DR, runbooks |
| Resilience | 07-04 | services/common/resilience.py: timeouts, retries with jitter, circuit breaker, bulkhead, load shedding |
| Automation and orchestration | 07-05 | Ansible, k8s/orders-deployment.yaml, HPA, canary, GitOps, mesh |
| Testing and chaos | 07-06 | Testcontainers, contracts, k6 tests/load/harvest.js, Toxiproxy, Chaos Mesh |
| Microservices | 08-01 | Bounded contexts, services/inventory/api.py, products_view (CQRS), versioning /v1→/v2, ADR-001 |
| Real time | 08-02 | Mosquitto with ACL, van_mqtt.py, mqtt_kafka_bridge.py, ws_server.py, tracking.js, SSE alerts |
| Cloud | 08-03 | infra/aws/main.tf (VPC across 3 AZs, EKS, RDS Multi-AZ, MSK, ElastiCache, S3, IRSA), FinOps |
| Serverless and edge | 08-04 | serverless/thumbnails, serverless/invoices, template.yaml, saga in Step Functions, CDN Worker, edge/van/aggregator.py |
| End to end | 08-05 | This lesson |
- The journey of one of Anna's orders end to end
Anna, in Valencia, buys two aged-cheese from Montblanc Dairy and one crianza-wine from Roble Alto Winery. The order is P-2026-000125. This is the complete journey, with the lesson in which each step was built.
sequenceDiagram
autonumber
participant A as Anna's browser
participant CDN as CDN + Worker
participant K as Kong
participant P as orders
participant I as inventory
participant PG as payments
participant KF as Kafka
participant C as catalog
participant AN as analytics (Flink, lake)
participant R as delivery
participant L as Lambda invoices
A->>CDN: GET /api/v1/catalog/products/aged-cheese (JWT)
CDN-->>A: HIT: listing for market valencia (08-04)
A->>CDN: POST /api/v1/orders (JWT, Idempotency-Key)
CDN->>K: verifies JWT signature, forwards (not cacheable)
K->>K: jwt plugin, rate-limiting, X-Request-Id (06-05)
K->>P: POST /orders + X-Request-Id
P->>P: Idempotency-Key seen before? no → create order (created) in Cassandra (04-04)
P->>P: start saga: row in sagas (03-05)
P->>I: gRPC ReserveStock (mTLS, ServiceInterceptor, 300 ms timeout, circuit breaker)
I->>I: SELECT FOR UPDATE on RDS, 15 min reservation; outbox: stock.reserved, stock.updated
I-->>P: Reservation OK
P->>PG: gRPC Charge (Idempotency-Key charge-P-2026-000125)
PG->>PG: external gateway (ACL), 1.2 s
PG-->>P: payment authorized
P->>P: status paid; outbox: order.created, payment.confirmed (same transaction)
P-->>K: 201 Created {id, status: paid}
K-->>CDN: 201 Created
CDN-->>A: 201 (p99 < 500 ms excluding the gateway: SLO 07-01)
Note over P,KF: outbox relay → orders.events (partitioned by order id)
KF-->>C: stock.updated → products_view + invalidate Redis (08-01, 04-05)
KF-->>AN: order.created, payment.confirmed → lake /km0/events; Flink stock_alerts (05-04)
KF-->>R: payment.confirmed → assign van-3, publish delivery.assigned
KF-->>L: payment.confirmed (group invoices-lambda) → PDF in km0-invoices (08-04)
A->>R: wss://.../ws {jwt}, {subscribe: P-2026-000125}
R->>R: authorize sub = customer; channel van:van-3
loop every 5 s
Note over R: van-3 → MQTT → bridge → delivery.positions → Redis pub/sub
R-->>A: {type: position, seq, lat, lon}
end
Five observations about the diagram that sum up the course:
- Only two synchronous calls between services (steps 9 and 12), both with a timeout, a circuit breaker and mTLS, and both because their response governs the next decision (08-01). Everything else is events.
- No transaction crosses services. The order is created in Cassandra, the reservation in RDS and the charge at the gateway as local transactions; the saga chains and compensates them (03-05); the outbox guarantees that events go out if and only if the local transaction commits (02-05).
- Idempotency appears four times: Anna's
Idempotency-Keyat Kong/orders(a double click does not create two orders), thereservation_idininventory, thecharge-<order>key towards the gateway, and theevent_idin every consumer. A retry at any point is safe. - Anna receives her response at step 17, before the catalogue view, the delivery assignment or the invoice exist. What she sees afterwards (the van's position, the invoice in her account area) arrives by asynchronous paths with seconds of delay, which is the normal state (08-01).
- The edge and the gateway do their job before anything reaches the services: the JWT is verified three times (Worker, Kong, service), rate limiting protects
orders, and the product listing never even reached the region.
- What the order leaves in its wake: traces, metrics and audit
An order does not only produce state; it produces evidence. What remains after P-2026-000125:
| Signal | Where | Content | Lesson |
|---|---|---|---|
| Trace | Tempo | A trace_id generated at Kong (or propagated from the browser), with spans: kong → orders POST /orders → inventory.ReserveStock (12 ms) → payments.Charge (1,210 ms, with child span gateway) → cassandra.insert → outbox.insert. Afterwards, linked spans (not children) in the consumers: catalog.projection, delivery.assign, invoices.handler, joined by the event_id propagated in the Kafka headers |
07-02 |
| Logs | Loki | JSON lines with request_id, trace_id, order_id, sub=u-anna, level and message, in every service; queryable with {service="orders"} | json | order_id="P-2026-000125" |
07-02 |
| Metrics | Prometheus | km0_orders_total{status="confirmed"} +1; latency histogram for POST /orders (0.5 s bucket); km0_grpc_client_duration_seconds{method="ReserveStock"}; km0_circuit_state{dependency="payments"}=0; lag of the catalog-projection group; km0_ws_connections +1 |
07-01 |
| Audit | audit.events → km0-audit (immutable, with legal retention) |
{"sub": "u-anna", "action": "orders.create", "resource": "P-2026-000125", "remote": ..., "request_id": ..., "result": "ok"}; and the access of orders-service to inventory recorded by the ServiceInterceptor |
06-05 |
| Business events | orders.events (7-day retention) and lake /km0/events/orders.events/date=2026-09-15/ (forever) |
order.created, stock.reserved, stock.updated, payment.confirmed, delivery.assigned |
02-04, 04-02 |
| State | Cassandra (orders_by_customer, orders_by_id, sagas), RDS (skus, reservations, outbox), products_view, S3 (invoices/u-anna/P-2026-000125.pdf) |
The order, the reservation, the completed saga, the view and the invoice | Modules 3 and 4 |
| Cost | Provider bill, by tag | A few fractions of a cent: invocations, transitions, GB-s, invoice egress | 08-03 |
The proof that observability has been done well is that with the order_id everything can be reconstructed: from the logs to the trace_id, from the trace to each service and its latency, from the metric to whether that order was one of those that broke the SLO, from the event in the lake to what analytics computed, from the audit record to who did it. That is what exercise 2 puts to the test.
- The architecture decisions as ADRs
In 08-01 the full ADR-001 was written. These are Kilometre Zero's six structural decisions in summary format: each with the alternative that was rejected and the negative consequence that was accepted, which is the part that in a year's time will tell whether it is still valid.
| ADR | Decision | Rejected alternative and why | Accepted consequence | Lesson |
|---|---|---|---|---|
| ADR-001 | Extract inventory from the monolith as a bounded context with km0_inventory (PostgreSQL with Patroni / RDS Multi-AZ) and a gRPC API + events |
Keep stock inside orders (would couple rules and deployments); stock in Cassandra (lightweight transactions expensive for compare-and-set); 2PC (locks and an external gateway outside the protocol) |
Eventual consistency between real stock and catalogue (a window of seconds); one more synchronous call on the critical path (+12 ms p99); a stateful component to operate | 08-01 |
| ADR-002 | Cassandra (3 nodes, RF=3, QUORUM) for km0_orders, with one table per query |
PostgreSQL with Citus (a coordinator and distributed transactions that the access pattern does not need); MongoDB (single primary with failover); PostgreSQL and accepting 30 s of unavailability during a campaign | No JOIN or ad hoc queries: every new question is a table or a CQRS view; consistency level chosen on every operation (a silent bug if chosen badly); operating compactions, repairs and snapshots |
03-02, 04-04 |
| ADR-003 | Orchestrated sagas from orders (order_saga.py, sagas table) to confirm an order |
Pure choreography (each service reacts to events: more decoupled, but the flow is implicit and hard to follow and to compensate in order); Step Functions (latency and cost per transition on a critical path of 1.2 M/month; lock-in) | orders knows about inventory and payments (coupling accepted); the orchestrator is a stateful component that must be recovered on start-up; no isolation between sagas (countermeasures from 03-05: reservations with expiry, pending states) |
03-05, 08-04 |
| ADR-004 | Kafka as the single event bus (orders.events keyed by order, delivery.*, inventory.alerts, audit.events), with an outbox in the producers and idempotent consumers |
RabbitMQ for everything (no retention or replay, which are essential for rebuilding CQRS views and feeding the lake); HTTP calls between services to notify (temporal coupling, symptom 2); an ESB with transformations (the lesson of SOA) | Eventual consistency as the norm; ordering only within a partition; event schemas as contracts to version; one critical cluster (MSK) on which everything depends; finite retention except in the lake | 02-04, 02-05, 08-01 |
| ADR-005 | Kubernetes (EKS) with a service mesh, GitOps and canaries as the deployment platform for the services; managed services for data (RDS, MSK, ElastiCache, S3) | Virtual machines with Ansible for everything (no autoscaling or reconciliation, manual deployments, symptom 3); a PaaS such as Cloud Run/App Runner per service (less control over networking, mesh and operators; it would fit a small version); self-managed Kubernetes (operating the control plane for no benefit) | Operational complexity (namespaces, RBAC, mesh, operators) taken on by a platform team; 1-2 ms per sidecar hop; selective lock-in on the managed services; a permanent baseline cost (≈ €3,900/month with reservations) | 07-05, 08-03 |
| ADR-006 | Serverless (Lambda with SAM) for sporadic event tasks: thumbnails, invoices, gateway webhooks, scheduled tasks; Step Functions only for long, rare workflows (producer offboarding) | Consumers on Kubernetes for everything (permanent cost for tasks that run for seconds a day; probes and on-call for no reason); serverless for orders or the WebSocket (latency, state, connections); Step Functions for the order saga (ADR-003) |
High lock-in in the adapters (limited by separating handler and logic); automatic retries that force strict idempotency; cold starts; observability that has to be joined to the cluster's; revisit the cost if the frequency multiplies by ten | 08-04 |
Every ADR lives in km0/docs/adr/ADR-00N-*.md, in the full format from 08-01. Note that three of the six decisions (002, 003, 006) explicitly reject a "more modern" or "more managed" option: the architecture is not the sum of the best technologies but of the decisions that answer a symptom at a known cost.
- The five symptoms of 01-06, revisited
| Symptom (01-06) | What it demanded | How it was resolved | How it is verified |
|---|---|---|---|
| 1. Campaign spikes take everything down (Grape Harvest ×15; 92% catalogue reads) | Scale the catalogue separately, serve reads from a cache and photos from elsewhere, isolate failures | catalog as its own service with Redis cache-aside (04-05), photos in S3 with thumbnails by Lambda (04-03, 08-04) and a CDN with immutable (08-04); HPA and spot nodes during campaigns (07-05, 08-03); rate limiting at Kong (06-05); load shedding (07-04) |
k6 tests/load/harvest.js at ×15 in staging before every campaign (07-06): catalogue p99 < 100 ms, CDN hit ratio > 95%, orders without degradation |
| 2. A payments failure takes everything down (gateway at 30 s; connections exhausted) | payments isolated with its own resources, slowness contained, confirmation without a global transaction |
payments with an ACL in front of the gateway (08-01); timeout, circuit breaker and bulkhead on orders → payments (07-04); saga with expiring stock reservation and a "payment pending" state (03-05); its own database (ADR-001/002) |
Chaos experiment: Toxiproxy adds 30 s of latency to the gateway (07-06); the circuit opens in < 10 s, catalogue and tracking carry on, orders stay pending and complete when the circuit closes |
| 3. Teams step on each other when deploying (4 teams, Tuesday-and-Thursday deployment) | Independent deployment units per domain, with no downtime | One Deployment per service with rolling updates and automatic canaries against the SLO (07-05); contract tests that protect consumers (07-06); GitOps: deploying is merging; teams owning services and on-call per service (08-01) | Deployment frequency per service (DORA metric): delivery several times a day, payments once per sprint, with no coordination; zero coordinated deployments in the last quarter |
| 4. Real-time courier telemetry (140 couriers, 2.4 M positions/day in the orders table; customers asking) | A channel for high-frequency streams, bidirectional communication, time-series storage | MQTT with QoS 1, ACL and sessions (08-02); bridge to delivery.positions with a key (02-04); Cassandra for positions (04-04); Flink for the dashboard (05-04); WebSockets with fan-out via Redis to Anna and to the operators (08-02); aggregator on the van (08-04) |
End-to-end latency (van → browser) p95 < 3 s; no position writes to the orders database; km0_ws_drops_total ≈ 0 outside incidents |
| 5. Analytics chokes production (3 h of nightly queries against production) | Separate the analytical load with a copy fed by events | analytics consumes orders.events and delivery.* into the lake (04-02); Spark daily_sales.py on ephemeral EMR with spot (05-03, 08-03); Flink for what cannot wait for the batch (05-04); Airflow km0_daily_sales (05-05); a separate km0_analytics |
Zero queries from analytics against km0_inventory or km0_orders (network policy and credentials prevent them: 06-04, 07-05); the DAG finishes before 06:00 with an SLA in Airflow; producers have forecasts they previously did not dare to ask for |
- Assessment against the course's criteria
A good course leaves criteria, not just techniques. These are the five with which Kilometre Zero assesses itself, and with which the student can assess any other platform.
The eight fallacies (01-04)
| Fallacy | Where the platform assumes it is false |
|---|---|
| The network is reliable | Timeouts on every call (07-04); retries with idempotency (02-05); QoS 1 and a persistent queue on the van (08-02, 08-04); replication and quorums (03-04) |
| Latency is zero | At most two synchronous calls on the critical path (08-01); caching at three tiers (04-05); CDN and Worker (08-04); measured latency SLO (07-01) |
| Bandwidth is infinite | Binary gRPC (02-03); MQTT with a 2-byte header (08-02); coalescing towards slow clients (08-02); leg aggregation on the van and a CDN for egress (08-04) |
| The network is secure | TLS everywhere (06-02); mTLS with the mesh (06-04); JWT verified at the edge, the gateway and the service (06-01, 06-05, 08-04); ACL on the broker; security groups and private subnets (08-03) |
| Topology doesn't change | Service discovery via Kubernetes and the mesh (07-05); consistent hashing (04-01); reconnection with backoff in clients (08-02); RDS failover via DNS (08-03) |
| There is one administrator | Teams owning services, an internal platform, Kubernetes RBAC and IAM per service account (08-01, 07-05, 08-03); contracts and ADRs as agreements |
| Transport cost is zero | Egress, cross-AZ traffic and NAT in the budget; VPC endpoints; client.rack; FinOps (08-03) |
| The network is homogeneous | Protobuf contracts and versioned event schemas (02-03, 02-05); the last mile designed for mobile networks (08-02) |
Consistency model per piece of data (03-01, 03-02)
| Data | Model | Mechanism | Who reads it that way |
|---|---|---|---|
| Available units of a SKU per market | Strong (linearizable per row) | SELECT FOR UPDATE on the RDS primary; synchronous replica |
inventory.ReserveStock |
| Charge status | Strong in payments; idempotent towards the gateway |
Local transaction + Idempotency-Key |
The saga |
| Anna's order (by id) | Quorum (read-your-own-writes guaranteed by QUORUM+QUORUM) |
Cassandra RF=3 | orders on confirmation; Anna when checking her order |
| A customer's order list | Eventual (ONE) |
Cassandra | The "my orders" screen |
| Availability on the catalogue listing | Eventual (seconds) | products_view via events + Redis + CDN 60 s |
Anna browsing |
| Producer dashboard | Eventual (seconds) | CQRS projection | Martha Hill |
| Van position | Eventual, "last wins" by seq |
MQTT retained, Redis last:, deduplication by sequence |
Anna, operators |
| Physical stock at the market stall | Multi-leader with local authority; CRDTs for counters | Reconciliation on sync | The Lleida terminal |
| Sales reports | Eventual (hours) | Lake + nightly Spark | Producers, management |
| Audit | Strong on write, immutable | Outbox → audit.events → S3 with object lock |
Compliance |
The lesson of this table is that "strong" appears twice and "eventual" six times: strong consistency is paid for and reserved for what truly needs it.
SLOs (07-01) and RPO/RTO (07-03)
| Service / data | SLO | RPO | RTO |
|---|---|---|---|
orders (create) |
99.9% availability; p99 < 500 ms (excluding the gateway) | km0_orders: 0 with QUORUM for a single node; minutes on loss of a region (snapshots) |
Node: 0 (no failover); region: 1-2 h (pilot light runbook) |
inventory |
99.95%; p99 < 100 ms on ReserveStock |
0 (synchronous Multi-AZ replica); minutes across regions | ~60 s (RDS failover); region: 30-60 min |
catalog |
99.9%; p99 < 200 ms (100 ms with the CDN) | N/A (rebuildable from events and S3) | Minutes (re-read orders.events) |
delivery real time |
99%; end-to-end latency p95 < 3 s | Positions: losses are tolerated | Minutes |
| Kafka | 99.95% | 0 with acks=all, min.insync.replicas=2 |
Seconds (leader election) |
analytics |
Daily report before 06:00 (Airflow SLA) | Hours | Hours (re-run the DAG) |
| Photos and invoices | 99.99% (S3) | ~0 (versioning + CRR) | Minutes (switch region at the CDN) |
Security controls (Module 6): who can do what
| Actor | Identity | Can | Cannot | Control |
|---|---|---|---|---|
| Anna (customer) | Keycloak JWT, roles: [customer], client web-km0 |
View the catalogue, create her own orders, view her orders and invoices, track her deliveries over WebSocket, chat with her producers | View Mark's orders; subscribe to a whole market; publish products | RBAC in services (06-01); authorization per subscription (08-02); rate limit per consumer (06-05) |
| Martha Hill (producer) | JWT, roles: [producer], claim producer_id: montblanc-dairy |
Publish and edit her products, upload photos via presigned URL, view her orders dashboard, receive stock alerts over SSE, reply in the chat | Touch La Vega Farm's products; see customers' personal data beyond name and delivery | ABAC "her products" in catalog (06-01); presigned URL with prefix (04-03); projection filtered by producer (08-01) |
| Jordan Hall (operator) | JWT, roles: [operator] |
Delivery dashboard for every market, reassign vans, send commands over MQTT (via delivery), view the delivery audit trail |
Modify prices; access payment data | RBAC; delivery publishes on commands with the bridge identity (08-02); audit of every command |
van-3 (device) |
Its own MQTT user, credential in Vault, revocable | Publish on km0/delivery/van-3/{position,status,leg}, read its commands |
Publish as van-7; read anything else |
Mosquitto ACL (08-02); validation at the bridge; local encryption (08-04) |
orders (service) |
mTLS certificate issued by the mesh (SPIFFE), client orders-service in Keycloak, service account with an IAM role |
ReserveStock, GetStock on inventory; Charge on payments; produce on orders.events; read its own secrets |
Read km0_inventory directly; produce on delivery.*; access S3 |
ServiceInterceptor per method (06-04); mesh and network authorization policies (07-05); IRSA (08-03); dynamic Vault credentials |
| Lambda invoices | The function's IAM role | Consume orders.events (own group), write to km0-invoices, PutItem on km0-idempotency |
Read km0-photos; write to Kafka |
SAM policies (08-04) |
| CDN Worker | No secret credential | Verify signatures with the public key, cache public responses |
Access databases; see the private key | "Nothing secret at the edge" design (08-04) |
| Platform team | IAM with MFA, Kubernetes RBAC per namespace | Operate the infrastructure, apply Terraform from the pipeline | Read decrypted payment data; apply Terraform from a laptop against production | Least privilege, GitOps, CloudTrail audit (08-03) |
- What is left out and next steps
No architecture is ever finished; this one has five open fronts, each with the course module that provides the tools to tackle it:
- Active-active multi-region. Today there is a pilot light (07-03, 08-03): the secondary region takes 1-2 hours to serve. Moving to active-active would require deciding the write model per piece of data:
km0_ordersin multi-DC Cassandra (LOCAL_QUORUMper region, 04-04) is natural; stock (inventory) is not, and would force partitioning authority by market (Girona and Lleida write ineu-west-1, Valencia ineu-central-1) with the multi-leader model of 03-04 for the markets split across regions. It is a months-long project and the pending ADR-007. - Full event sourcing. Today events are notifications of state changes that are stored separately (and forever in the lake). Event sourcing would make the events the source of truth for
orders: state is rebuilt by replayingorder.created,payment.confirmed, ... You gain perfect auditability and the ability to rebuild any view; you pay with snapshots, versioning of historical events and queries that always go through a projection. CQRS (08-01) has already prepared the ground. - Data mesh.
analyticsis today a central team that consumes everything. As the domains grow, each team would start publishing its data as a product (with a schema, quality, an SLA and an owner) andanalyticswould become a platform. It is Conway's law (08-01) applied to data. - The internal platform as a product. The platform team from 08-01 exists; what is missing is treating it as a product: a portal where a team creates a new service with its Deployment, its dashboards, its alerts, its pipeline and its template ADR in minutes (exercise 1 puts this to the test).
- Cost as a design constraint. FinOps (08-03) measures; the next step is for cost to enter architecture decisions the way latency does: euros per order as an SLO, a quarterly review of managed versus self-run services, and automatic shutdown of anything not serving traffic.
- The project: a reduced Kilometre Zero in
docker-compose
docker-composeThe course has shown code in every lesson; the project proposes building a reduced, working version on your laptop, with what fits in docker-compose and no cloud: catalog, orders, inventory, Kafka, PostgreSQL, Redis and Prometheus. payments is simulated, delivery and analytics are optional, and the rest of the platform (Keycloak, Kubernetes, Terraform, Lambda, CDN) is outside the minimum scope. What matters is not the size but that every milestone is verifiable with a command.
Final structure of the km0/ repository
km0/
├── README.md
├── docker-compose.yml # PostgreSQL, Kafka (KRaft), Redis, Prometheus, Grafana, the services
├── docs/
│ └── adr/ # ADR-001 … ADR-006
├── contracts/
│ ├── inventory.proto # ReserveStock, GetStock, WatchChanges (02-03)
│ └── events/ # JSON schemas for order.created, stock.updated, ... (02-05)
├── services/
│ ├── common/ # metrics.py, logs.py, traces.py, resilience.py, jwt.py
│ ├── catalog/ # api.py, cache.py (04-05), stock_projection.py (08-01)
│ ├── orders/ # http_api.py (v1/v2), order_saga.py (03-05), outbox_relay.py, repository.py
│ ├── inventory/ # api.py (08-01), grpc_server.py, _domain.py, _repository.py, _events.py
│ ├── payments/ # simulated_gateway.py, consumer.py
│ ├── delivery/ # van_mqtt.py, mqtt_kafka_bridge.py, ws_server.py, ws_kafka_consumer.py
│ └── analytics/ # lake_consumer.py
├── sql/
│ ├── inventory/ # skus, reservations, outbox, processed_messages
│ ├── catalog/ # products, products_view.sql
│ └── orders/ # sagas (in the reduced version, PostgreSQL instead of Cassandra)
├── simulations/ # clocks, partitions, quorums (Modules 1 and 3)
├── dags/ # km0_daily_sales.py (05-05)
├── serverless/ # thumbnails/, invoices/, template.yaml, saga/ (08-04)
├── edge/
│ ├── mosquitto/ # mosquitto.conf, acl, passwd (08-02)
│ ├── web/tracking.js # WebSocket client (08-02)
│ ├── worker/catalog.js # CDN Worker (08-04)
│ └── van/aggregator.py # (08-04)
├── certs/ # km0-ca and development certificates (06-04)
├── observability/
│ ├── prometheus.yml, alerts.yml # (07-01)
│ └── grafana/dashboards/ # RED per service, saga, consumer lag
├── k8s/ # orders-deployment.yaml, HPA, PDB, NetworkPolicy, overlays/ (07-05)
├── ansible/ # inventory and Kafka/Cassandra playbooks (07-05)
├── infra/aws/ # main.tf, variables.tf, outputs.tf, environments/ (08-03)
└── tests/
├── integration/ # Testcontainers: saga, idempotency, projection (07-06)
├── contract/ # Pact, protobuf and event compatibility
├── load/harvest.js # k6
└── chaos/ # Toxiproxy, Chaos Mesh manifestsdocker-compose.yml for the reduced version
# km0/docker-compose.yml — reduced version for the final project
services:
postgres:
image: postgres:16
environment: { POSTGRES_USER: km0, POSTGRES_PASSWORD: km0, POSTGRES_DB: km0 }
volumes:
- ./sql:/docker-entrypoint-initdb.d:ro # creates km0_inventory, km0_catalog, km0_orders and their tables
ports: ["5432:5432"]
healthcheck: { test: ["CMD-SHELL", "pg_isready -U km0"], interval: 5s, retries: 10 }
kafka:
image: apache/kafka:3.7.0 # KRaft: no ZooKeeper (03-03)
environment:
KAFKA_NODE_ID: 1
KAFKA_PROCESS_ROLES: broker,controller
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false" # topics are created explicitly (02-04)
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
ports: ["9092:9092"]
kafka-init: # creates the topics and exits
image: apache/kafka:3.7.0
depends_on: [kafka]
entrypoint: ["/bin/sh", "-c"]
command: |
"sleep 5 &&
/opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 --create --if-not-exists --topic orders.events --partitions 6 --replication-factor 1 &&
/opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 --create --if-not-exists --topic inventory.alerts --partitions 3 --replication-factor 1 &&
/opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 --create --if-not-exists --topic orders.events.dlq --partitions 1 --replication-factor 1"
redis:
image: redis:7
ports: ["6379:6379"]
inventory:
build: ./services/inventory
environment: { PG_DSN: "postgresql://km0:km0@postgres/km0_inventory", KAFKA_BOOTSTRAP: "kafka:9092" }
depends_on: { postgres: { condition: service_healthy } }
ports: ["50051:50051", "9101:9100"] # gRPC and metrics
orders:
build: ./services/orders
environment:
PG_DSN: "postgresql://km0:km0@postgres/km0_orders"
KAFKA_BOOTSTRAP: "kafka:9092"
INVENTORY_GRPC: "inventory:50051"
PAYMENTS_URL: "http://payments:8002"
depends_on: [inventory, kafka-init]
ports: ["8001:8000", "9102:9100"]
payments:
build: ./services/payments # simulated gateway: rejects 5% and takes 200-1500 ms
ports: ["8002:8002"]
catalog:
build: ./services/catalog
environment:
PG_DSN: "postgresql://km0:km0@postgres/km0_catalog"
KAFKA_BOOTSTRAP: "kafka:9092"
REDIS_URL: "redis://redis:6379"
depends_on: [redis, kafka-init]
ports: ["8003:8000", "9103:9100"]
prometheus:
image: prom/prometheus:v2.53.0
volumes: ["./observability/prometheus.yml:/etc/prometheus/prometheus.yml:ro"]
ports: ["9090:9090"]
grafana:
image: grafana/grafana:11.1.0
volumes: ["./observability/grafana:/etc/grafana/provisioning:ro"]
ports: ["3000:3000"]Verifiable milestones
| Milestone | What to build | How to check it |
|---|---|---|
| M1. Start-up | docker compose up brings everything up; the three services expose /health and /metrics |
curl localhost:8001/health → {"status":"ok"}; curl localhost:9090/api/v1/targets shows all three up |
| M2. Catalogue with cache | GET /products/aged-cheese with cache-aside in Redis (04-05) |
First request > 5 ms and X-Cache: MISS; second < 1 ms and HIT; redis-cli keys 'product:*' shows the key with a TTL |
| M3. Stock reservation over gRPC | inventory.ReserveStock with SELECT FOR UPDATE, idempotent by reservation_id, publishing to outbox (02-03, 08-01) |
grpcurl -plaintext -d '{"reservation_id":"r1","order_id":"P-1","market":"girona","lines":[{"product":"aged-cheese","units":2}]}' localhost:50051 km0.inventory.v1.Inventory/ReserveStock twice → same response, stock deducted once; SELECT * FROM outbox with stock.reserved and stock.updated |
| M4. Outbox → Kafka | Relay that reads outbox and produces to orders.events keyed by order, marking them as published (02-05) |
kafka-console-consumer --topic orders.events --from-beginning --property print.key=true shows the events with their envelope; killing the relay halfway and restarting does not duplicate (unique event_ids) |
| M5. Create order with saga | POST /orders with Idempotency-Key; saga reserve → charge → confirm; compensation if payments rejects (03-05) |
curl -X POST -H 'Idempotency-Key: k1' ... twice → a single order; with the gateway forced to reject (PAYMENTS_MODE=reject), the order ends up rejected and the stock comes back (SELECT available FROM skus); SELECT * FROM sagas shows the transitions |
| M6. CQRS projection | catalog consumes stock.updated, updates products_view and invalidates Redis (08-01) |
After M5, GET /products/aged-cheese shows the updated available in < 2 s; SELECT * FROM processed_messages grows; re-sending an event by hand with the same event_id changes nothing |
| M7. Resilience | Timeouts, retries with jitter and a circuit breaker on orders → payments (07-04) |
Stop payments (docker compose stop payments): requests fail fast (< 1 s, not 30 s) once the circuit opens; km0_circuit_state{dependency="payments"} = 2 in Prometheus; when payments comes back, it closes by itself |
| M8. Observability | RED per service, saga latency, consumer lag; a Grafana dashboard; an SLO alert (07-01) | Dashboard with rate(km0_orders_total[1m]) by status and p99 of POST /orders; the alert fires when 5% errors are provoked with PAYMENTS_MODE=errors |
| M9. Load | k6 with 50 virtual users creating orders and reading the catalogue for 2 minutes (07-06) | Catalogue p99 < 50 ms; no duplicate orders (SELECT count(*), count(DISTINCT idempotency_key) equal); final stock = initial − sold |
| M10. Integration test | Testcontainers bringing up PostgreSQL and Kafka; a test of the saga with a rejecting payments and of consumer idempotency (07-06) |
pytest tests/integration green in under 2 minutes |
| Optional A. Real time | Mosquitto + mqtt_kafka_bridge.py + ws_server.py + tracking.js (08-02) |
Publish with mosquitto_pub -t km0/delivery/van-3/position -q 1 -m '{...}' and see it in the browser; publish a lower seq and check that it is discarded |
| Optional B. Analytics | A consumer that writes events to Parquet files by date; a local Spark script that computes sales per producer (05-03) | spark-submit daily_sales.py --date 2026-09-15 produces the table with Montblanc Dairy at the top |
Every milestone should end with a short ADR if a decision is taken (for example, "PostgreSQL instead of Cassandra for orders in the reduced version: context, consequences") and with the tests that verify it in tests/. The result is a repository you can show and talk about in an interview with the same precision with which this course has talked about Kilometre Zero.
- Study guide
The course has been a map; these are the territories to explore next. Works and papers are cited by title and author, without links: all of them are easy to find by name.
Reference books
- Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017). The book that most resembles Modules 2, 3 and 4 of this course, in far greater depth: replication, partitioning, transactions, consensus, batch and stream processing. It is the natural next read.
- Andrew S. Tanenbaum and Maarten van Steen, Distributed Systems (3rd ed., available free of charge from the authors). The classic academic text: models, communication, synchronisation, consistency and fault tolerance with formal rigour. It complements Modules 1 and 3.
- Sam Newman, Building Microservices (2nd ed., O'Reilly, 2021) and Monolith to Microservices (2019). Everything in 08-01 in detail: boundaries, migration, testing, organisation.
- Betsy Beyer et al. (eds.), Site Reliability Engineering: How Google Runs Production Systems and The Site Reliability Workbook. The origin of the SLOs, error budgets, postmortems and on-call of Module 7. Available free online from Google.
- Eric Evans, Domain-Driven Design (2003), and Vaughn Vernon, Implementing Domain-Driven Design (2013), to go deeper into the bounded contexts of 08-01.
- Gregor Hohpe and Bobby Woolf, Enterprise Integration Patterns (2003): the catalogue of messaging patterns from which 02-04 and 02-05 took their names.
- Neal Ford, Mark Richards et al., Software Architecture: The Hard Parts (2021): trade-offs of decomposition, distributed data and sagas, in the same ADR spirit.
Foundational papers (all findable by title)
- Leslie Lamport, "Time, Clocks, and the Ordering of Events in a Distributed System" (1978): the basis of 01-05. And "The Part-Time Parliament" (1998) and "Paxos Made Simple" (2001) for consensus.
- Diego Ongaro and John Ousterhout, "In Search of an Understandable Consensus Algorithm" (Raft, 2014): what etcd and Kafka KRaft implement (03-03).
- Giuseppe DeCandia et al., "Dynamo: Amazon's Highly Available Key-value Store" (2007): consistent hashing, quorums, vector clocks and the choice of availability that Cassandra inherits (04-01, 04-04).
- Jeffrey Dean and Sanjay Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters" (2004), and Sanjay Ghemawat et al., "The Google File System" (2003): 05-02 and 04-02.
- James C. Corbett et al., "Spanner: Google's Globally-Distributed Database" (2012): strong consistency at global scale with bounded clocks (TrueTime), the counterpoint to Dynamo.
- Jay Kreps, Neha Narkhede and Jun Rao, "Kafka: a Distributed Messaging System for Log Processing" (2011), and Jay Kreps's essay "The Log: What every software engineer should know about real-time data's unifying abstraction" (2013): the idea of the log as the backbone (ADR-004).
- Hector Garcia-Molina and Kenneth Salem, "Sagas" (1987): the original paper behind 03-05.
- Seth Gilbert and Nancy Lynch, "Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services" (2002), and Daniel Abadi, "Consistency Tradeoffs in Modern Distributed Database System Design" (PACELC, 2012): 03-02.
- Marc Shapiro et al., "Conflict-free Replicated Data Types" (2011): the CRDTs of 03-01 and 08-04.
- Peter Deutsch and James Gosling, the "Fallacies of Distributed Computing", and Peter Bailis et al., "Highly Available Transactions: Virtues and Limitations" (2013), to continue with the fallacies and isolation models.
- Tyler Akidau et al., "The Dataflow Model" (2015): event time, watermarks and windows from 05-04.
Practice
- Jepsen (Kyle Kingsbury): public analyses of how real databases fail under partitions; the best school of scepticism about advertised guarantees.
- Michael Nygard's Release It! (2nd ed., 2018), for the stability patterns of 07-04 told through real incidents.
- Build the project from section 9 and then break it with the experiments from 07-06.
Common Mistakes and Tips
- Presenting the architecture as a diagram without decisions. The diagram in section 1 is worth little without the ADRs in section 5; what you learn from an architecture is its rejected alternatives and its accepted consequences.
- Assessing by technologies rather than by criteria. "It uses Kafka and Kubernetes" says nothing about whether it is well designed. The tables in section 7 (fallacies, consistency per piece of data, SLOs, RPO/RTO, who can do what) do.
- Starting the project with the infrastructure. Kubernetes, mesh and Terraform with no service to deploy is the mistake of "choosing technologies before problems" from 01-06. The project starts at M1 with three services and
docker-compose. - Believing the platform is finished. Section 8 lists five open fronts; an architecture without a backlog is one nobody is looking at.
- Tip: in any new system, start by writing down the symptoms (01-06), the consistency-per-data table (03-01) and the SLOs (07-01) before drawing boxes. They are the three pages that produce the most correct decisions.
- Tip: keep this course as a reference by lesson: when in a year's time you have to decide between orchestration and choreography, or how much TTL to put on a CDN, the corresponding lesson has the table.
Exercises
Exercise 1: bringing in the "reviews" domain
In 08-01 it was decided that reviews are a bounded context of their own with event-based communication. Now design it end to end applying the whole course: (a) the contract of events it consumes and produces (with the envelope from 02-04) and the data model with its consistency model (03-01) and its storage (Module 4); (b) the public API at Kong with authentication and authorization (who may review what; 06-01, 06-05) and how the average reaches the listing cached by the CDN (08-04); (c) SLO, metrics and alerts (07-01), and what traces and audit records a review leaves behind; (d) deployment (07-05), RPO/RTO (07-03) and what would have to be added to main.tf (08-03); (e) ADR-007 in summary format (decision, rejected alternative, consequence).
Exercise 2: an end-to-end incident
Saturday 19 September, 18:42, Artisan Cheese Week. The alert "orders SLO: error budget burn ×14 in 5 min" fires. Available data:
- Prometheus:
rate(km0_orders_total{status="rejected_stock"}[5m])goes from 0.3/s to 9/s at 18:40; the p99 latency ofPOST /ordersdrops from 420 ms to 95 ms;km0_grpc_client_duration_seconds{method="ReserveStock"}p99 = 8 ms;km0_circuit_state{dependency="inventory"}= 0; lag of thecatalog-projectiongroup = 0. inventorylogs (Loki): from 18:39:50, hundreds of lines{"level":"warning","msg":"InsufficientStock","product":"aged-cheese","market":"girona","available":0,"requested":2,"request_id":...}.cataloglogs:{"level":"info","msg":"view updated","product":"aged-cheese","market":"girona","applied":true}at 18:39:48, and none after that for that product.- Traces: the rejected orders have spans
kong → orders → inventory.ReserveStock (InsufficientStock)of 90 ms; none reachespayments. products_viewincatalog:aged-cheese / girona: available = true, approx_units = 0, stock_version = 1758300000123.- Audit: at 18:39:45,
{"sub":"mhill","action":"stock.adjust","resource":"aged-cheese/girona","from":140,"to":0,"source":"producer-dashboard"}.
(a) Reconstruct the complete causal chain. (b) Is it a platform incident, a product incident or an operations incident? Which part of the behaviour is correct and which is a defect? (c) Locate the exact defect in the code from 08-01 (stock_projection.py or products_view.sql) and fix it. (d) Write the three postmortem actions (07-03) and say which signal would have detected the problem before the SLO did.
Exercise 3: trimming the architecture down for three people
A three-person startup wants to launch a marketplace like Kilometre Zero in a single market, with 20 producers, 300 orders a day and 6 couriers. They have three months and cannot operate Kubernetes or Kafka. Using the "when not to use microservices" table from 08-01 and the selective lock-in of 08-03, design the minimum architecture: (a) what stays as a modular monolith and with what internal structure; (b) which three Kilometre Zero decisions you would keep intact because they are cheap and prevent irreversible mistakes; (c) how you would handle real-time tracking and photos without Kafka, MQTT or Lambda; (d) which measurable signals would indicate that the time has come to extract the first service, and which one it would be.
Solutions
Exercise 1.
(a) It consumes order.delivered (published by delivery on orders.events with the standard envelope; data: {order_id, customer, lines: [{product, producer}], delivered_ms}) to know what can be reviewed. It produces on a new topic reviews.events (3 partitions, key = product, so that a product's reviews reach the projection in order): review.created {review_id, customer, order_id, product, producer, points, comment, timestamp_ms} and review.replied {review_id, producer, reply, timestamp_ms}. Model: tables reviewable (customer, order_id, product, delivered_ms, reviewed bool) and reviews (id, customer, product, producer, points, comment, reply, created_at) in its own PostgreSQL database km0_reviews (a small RDS, without Multi-AZ at first): low volume (one per delivered line), relational queries (by product, by producer, awaiting reply), and the rule "one review per line" requires a UNIQUE (customer, order_id, product) constraint with local strong consistency. The average on the listing is eventual (a projection in catalog), and the producer dashboard reads its own database with strong consistency.
(b) Routes at Kong: POST /api/v1/reviews (role customer), GET /api/v1/products/{slug}/reviews (public, cacheable for 60 s), POST /api/v1/reviews/{id}/reply (role producer), GET /api/v1/producers/me/reviews?pending=true (role producer). Authorization in the service: on creation, claims.sub must have an unreviewed row in reviewable for that order and product (ABAC on its own data, without calling orders); on reply, claims.producer_id must match reviews.producer. Specific rate limiting (5 reviews/minute per consumer) against abuse. The average reaches the listing because catalog consumes review.created and maintains points_sum and review_count in products_view (idempotent by event_id); the listing stays public, max-age=60 per market, and the average changes with up to a minute's delay at the CDN, which is acceptable. The three latest comments are served on the public reviews route (cacheable, with no personal data beyond the first name, which the customer consented to).
(c) SLO: 99.5% availability and p99 < 300 ms for creation (it is not on the purchase path: a looser SLO than orders). Metrics: km0_reviews_total{result}, latency per route, lag of the order.delivered consumer and of the catalog consumer on reviews.events. Alerts: error budget burn, lag > 5 min (a delivery that cannot be reviewed), and an anomalous rate of unauthorized (abuse). Trace: kong → reviews POST → postgres, linked to the catalog consumer by the event_id. Audit: reviews.create and reviews.reply on audit.events (producers' replies are public content, and moderation needs to know who wrote what).
(d) k8s/reviews-deployment.yaml copying the orders template (2 replicas, probes, resources, topologySpreadConstraints, HPA by CPU, a NetworkPolicy that only allows ingress from Kong and egress to its RDS and to Kafka, serviceAccountName with IRSA and no S3 permissions). RPO: minutes (RDS backups and PITR; losing reviews is tolerable but annoying); RTO: 1 hour (it does not block sales). In main.tf: aws_db_instance.reviews (db.t4g.small, no multi_az), its security group (5432 only from EKS), an IRSA module with no S3 policies, and the reviews.events topic managed with the kafka provider or created by the pipeline.
(e) ADR-007: "reviews as its own bounded context with PostgreSQL and communication exclusively through events". Rejected alternative: inside catalog (it would couple moderation to the catalogue's deployments during campaigns and mix two languages: "published product" and "review"). Accepted consequence: the listing shows the average with up to a minute's delay; one more component with a database, a dashboard and on-call (owned by the catalogue and producers team); if delivery stops publishing order.delivered, nobody can review until the lag recovers (alert).
Exercise 2.
(a) Causal chain: at 18:39:45, Martha Hill adjusts the stock of aged-cheese in Girona from 140 to 0 from her dashboard (probably a data-entry mistake, or a genuine withdrawal of the product). inventory applies the adjustment and publishes stock.updated {available: 0}. At 18:39:48, catalog receives the event and the projection applies it (applied: true) → approx_units = 0. But available stays true. From 18:39:50, customers who see the listing (with available = true) add aged-cheese to their basket and confirm; orders starts the saga, inventory answers InsufficientStock in 8 ms, the saga compensates and the order ends up rejected_stock. The p99 latency drops (rejected orders never go through the gateway), the rejection rate rises ×30 and the error budget is consumed because rejected_stock counts as a failure of the orders SLO.
(b) It is a mixed incident. The saga, the circuit and the compensation work correctly: no order was charged without stock, no reservation was left hanging. Martha's adjustment is a legitimate business action (and the audit records it precisely). The defect lies in the platform: the catalogue listing keeps saying "available" with 0 units, steering hundreds of customers into a certain rejection. It is a defect in the CQRS projection, not in eventual consistency: the delay was 3 seconds, which is acceptable; the problem is that the projected data is incorrect permanently.
(c) In stock_projection.py, apply computes avail = d["available"] >= AVAILABLE_THRESHOLD with AVAILABLE_THRESHOLD = 1: with available = 0, avail = False, and the UPDATE should have set available = FALSE... unless the stock_version < %(v)s condition failed. But the log says applied: true, so the UPDATE did affect the row. So the defect is not there; look at products_view.sql: available BOOLEAN NOT NULL DEFAULT FALSE and approx_units are updated together in the same UPDATE. The row shows approx_units = 0 and available = true, which is only possible if another write path set available = true afterwards: the product publication path used by the producer (catalog creates/updates the row when the producer edits the listing and, according to 08-01, "the row is created by catalog when the producer publishes the product"). Martha edited the listing (perhaps the same adjustment from the dashboard touches both product and stock), and the publication UPSERT overwrites available with its default value or with the value the listing had in the form, without respecting stock_version. The defect is that two paths write the same column with different rules: the fix is for listing publication to never write available or approx_units (columns owned exclusively by the projection), that is, INSERT ... ON CONFLICT (slug, market) DO UPDATE SET name = EXCLUDED.name, price_cents = EXCLUDED.price_cents, producer = EXCLUDED.producer without touching the stock columns; and, as a safety net, a CHECK (NOT available OR approx_units >= 1) on the table that would make the incoherent write fail rather than let it through. It is the "data owner" principle of 08-01 applied inside a service, column by column.
(d) Blameless postmortem (07-03): 1. Fix the publication UPSERT and add the CHECK; an integration test (07-06) that publishes the listing after a stock.updated to 0 and verifies available = false. 2. Add a confirmation in the producer dashboard for stock adjustments that change more than 50% ("Set aged-cheese in Girona to 0 units? There were 140"), because the audit shows an improbable jump. 3. Separate stock rejections (a correct business outcome) from platform errors in the orders SLO, or create a specific "rejected_stock rate per product" alert with a low threshold, so that the symptom is attributed properly. The signal that would have caught it earlier: a coherence alert on the projection, count(products_view where available and approx_units = 0) > 0, evaluated every minute (a cheap query), or the Flink stock_alerts metric from 05-04 for aged-cheese in Girona, which fired at 18:39:50 towards inventory.alerts and which Martha received over SSE (08-02) but which nobody correlated with the rejections.
Exercise 3.
(a) A modular monolith in Python (FastAPI) with one managed PostgreSQL database (RDS or Cloud SQL with automatic backups and PITR, without Multi-AZ at first), deployed on a container PaaS (Cloud Run, App Runner, Fly.io, Render: 2 instances, autoscaling included, managed TLS). Internal structure: the six packages from 01-06 (catalog, orders, inventory, payments, delivery, analytics), each with its api.py as the only public module (08-01), PostgreSQL schemas separated per package (inventory.skus, orders.orders) and a lint rule (import-linter) that forbids importing another package's private modules and SQL queries across schemas. The order confirmation transaction is local (a single database): reservation, order and charge record in one transaction, with the gateway call outside it (charge first with an Idempotency-Key, then confirm in the transaction; if the confirmation fails, refund). A small managed Redis for the catalogue cache and the rate limiter. A single repository, one pipeline, docker-compose for development.
(b) Three cheap decisions that prevent irreversible mistakes: 1. The event envelope and an events table (outbox) from day one, even if the "bus" is the database itself and the consumer a process inside the monolith: when Kafka arrives, the producers already publish and the consumers are already idempotent; and the analytical lake can start by exporting that table to Parquet every night. 2. Idempotency in the write APIs (Idempotency-Key on create order, deterministic keys towards the gateway): it is what in 02-05 and 08-04 prevents duplicate charges and orders, it costs one table, and it cannot be added later without migrating clients. 3. Minimal observability with an SLO: JSON logs with request_id, RED metrics and a written SLO for create order; without this, there will be no data to decide when to extract anything (part d). And a free fourth one: ADRs from day one.
(c) Tracking: 6 couriers sending positions via HTTPS POST every 10 s to the monolith (60 requests/minute: nothing), stored in a delivery.positions table with 24 h retention (at this volume, PostgreSQL is more than enough), and customers using SSE or polling every 5 s against an endpoint that returns the order's last position (with Cache-Control: max-age=4 so that the PaaS or a CDN absorbs repeats). No MQTT or WebSockets: there is no volume to justify them, and SSE with automatic reconnection covers the experience. Photos: upload via presigned URL to S3/GCS (04-03, kept) and thumbnails generated in the monolith itself as a background task (a simple queue in PostgreSQL with SELECT ... FOR UPDATE SKIP LOCKED, processed by a worker in the same deployment), with a free or low-cost CDN in front of the bucket using immutable keys (08-04, kept because it is almost free and avoids egress).
(d) Signals for extracting the first service, measurable thanks to (b)-3: the p99 of create order degrades when catalogue traffic rises (contention on the same database or CPU: symptom 1); the team grows from 3 to 8-10 people and catalogue deployments break orders (symptom 3); read traffic exceeds what Redis plus a PostgreSQL read replica can absorb; or the gateway's latency starts to block connections (symptom 2). The first to go, as in 01-06, is catalog (reads only, no cross-cutting transactions, rollback with a route change) or, if the dominant symptom is the gateway, payments with its ACL. Never orders and inventory first: they are the ones that demand sagas, and sagas are only justified when the boundary already exists for another reason. The rule from 08-01 holds in reverse: three people with no symptoms is exactly the row of the table that says "modular monolith".
Conclusion
Kilometre Zero began as a Python monolith with one database and five symptoms, and ends as a platform that can be drawn in a diagram, walked through with an order, justified with six ADRs and assessed against the criteria the course itself has provided: the eight fallacies assumed false at specific points of the design, a consistency-per-data table where "strong" appears twice and "eventual" six times, SLOs and RPO/RTO per component, and a who-can-do-what table from Anna to the CDN Worker. Each of the five symptoms from 01-06 has its resolution and its way of being verified; each decision has its rejected alternative and its price; and what is left out is written down, with the module that provides the tools to tackle it. The proposed project reduces all of that to what fits on a laptop, with ten milestones that are checked with a command, and the study guide points to the books and papers where every idea in this course has its origin and its depth.
What the student can now do is not "use Kafka" or "deploy on Kubernetes", although that too. It is something more durable: faced with a system that has to be spread across several machines, they know to write down the symptoms first and not the technologies; to cut boundaries where the language changes meaning and not where there is a table; to choose for each piece of data the consistency model it needs and pay only for that; to give every call a timeout, every message an identifier and every consumer idempotency, because the network is going to fail; to replace the global transaction with a saga that compensates; to put data where its workload asks for it, from PostgreSQL to Cassandra, from Redis to S3, from the CDN to a van's SQLite queue; to separate what is computed in batches from what cannot wait; to authenticate and authorize at every boundary without trusting any; to measure with SLOs, observe with traces, recover with tested backups and rehearse failures before they happen; to describe infrastructure as code and cost as a metric; and to leave every decision written down with its context and its consequences, so that whoever comes next can change it knowingly.
That is what designing distributed architectures means: not piling up pieces, but knowing for each one what is gained, what is lost and why it was chosen. Kilometre Zero has been the pretext; the judgement is what the student takes away. Thank you for following the course this far, and good luck with the next system that has to be spread across several machines: you now know where to start.
Distributed Architectures Course
Module 1: Introduction to Distributed Systems
- Basic Concepts of Distributed Systems
- Distributed System Models
- Advantages and Challenges of Distributed Systems
- The Fallacies of Distributed Computing
- Time, Clocks and Event Ordering
- From Monolith to Distributed Platform: the Kilometre Zero Case
Module 2: Communication in Distributed Systems
- Communication Protocols
- RPC and RMI
- gRPC and Data Serialization
- Messaging and Message Queues
- Asynchronous Communication Patterns
Module 3: Consistency and Replication
- Consistency Models
- The CAP Theorem and PACELC
- Consensus Algorithms
- Data Replication
- Distributed Transactions and Sagas
Module 4: Distributed Storage
- Data Partitioning and Consistent Hashing
- Distributed File Systems
- Object Storage
- Distributed Databases
- Distributed Caches
Module 5: Distributed Computing
- Distributed Computing Models
- MapReduce and Hadoop
- Spark and In-Memory Computing
- Stream Processing
- Job Scheduling and Data Pipelines
Module 6: Security in Distributed Systems
- Authentication and Authorization
- Encryption and Data Protection
- Identity Management
- Service-to-Service Security: mTLS and Secrets Management
- API Gateways, Rate Limiting and Auditing
Module 7: Monitoring and Maintenance
- Monitoring Distributed Systems
- Centralized Logs and Distributed Tracing
- Failure Management and Recovery
- Resilience Patterns: Timeouts, Retries and Circuit Breakers
- Automation and Orchestration
- Testing Distributed Systems and Chaos Engineering
