The previous lesson closed Module 7 with a question that has been open since Module 1: when the Python+PostgreSQL monolith was split into catalog, orders, inventory, payments, delivery and analytics, what made the result a system rather than a collection of six programs calling each other over the network? Seven modules have built the pieces (gRPC and Kafka, sagas and replicas, Cassandra and Redis, Spark and Flink, Keycloak and Vault, Prometheus and Kubernetes), but none of them has explained the criteria used to decide where to cut, which data each service owns, why catalog does not query the inventory database even though it technically could, or what each of those decisions costs. This lesson looks at Kilometre Zero from above, as a microservices architecture: what microservices are (and what they are not), how boundaries are drawn with a lightweight Domain-Driven Design, how queries that span services are answered when each service owns its own data, how to choose between synchronous and asynchronous communication, how to migrate away from a monolith without stopping the business, how teams are organised and, above all, when it is not worth doing. It ends with the format that keeps those decisions on record: the ADR. Real time (08-02), the cloud (08-03) and serverless (08-04) are left for the lessons that follow.

Contents

  1. What microservices are and what they are not
  2. The five defining characteristics
  3. How to cut the boundaries: lightweight Domain-Driven Design
  4. Cutting mistakes
  5. Data: one database per service and its consequences
  6. CQRS: a producer's orders dashboard and the products_view view
  7. Communication: synchronous, asynchronous, APIs, gateway, BFF and mesh
  8. Migration patterns: assessing the strangler plan
  9. Organisation: teams, Conway and the internal platform
  10. When not to use microservices and the return to the modular monolith
  11. Recording decisions: ADR-001
  12. Common Mistakes and Tips
  13. Exercises
  14. Conclusion

  1. What microservices are and what they are not

Lesson 01-02 introduced "services/microservices" as one more architectural model, alongside client-server, tiers and P2P, and deferred the details to this lesson. The working definition is this: a microservices architecture decomposes an application into small services, each aligned with a business capability, which are deployed independently and communicate over the network through explicit contracts. What sets it apart is neither size nor technology, but three mutually reinforcing properties: independent deployment, data ownership and boundaries drawn by the business. It is worth placing it next to its two neighbours, with which it is often confused.

Aspect Modular monolith Classic SOA Microservices
Deployment unit One (the whole process) A few, large, sharing a corporate bus (ESB) Many, small, one per capability
Boundaries Packages/modules within the same codebase; crossed with function calls Coarse-grained services, often one per system (CRM, ERP) By business capability (bounded context)
Data One shared database Databases shared between services; the ESB transforms One database per service; nothing shared
Communication In-process calls Centralised bus with orchestration and transformation logic "Dumb pipes, smart endpoints": HTTP/gRPC, events; the logic lives in the services
Consistency Local ACID transactions Distributed transactions (2PC, 03-05) Eventual consistency, sagas
Teams One or several on the same codebase Per system or per tier One team owning each service, end to end
Scaling All or nothing (copies of the process) Per service, but with the ESB as the bottleneck Per service, independently
Failure One error takes down the whole process The ESB is a single point of failure Isolated per service (when done well)
When it fits Small teams, a domain that is still unstable, latency-critical Integrating heterogeneous legacy systems Several teams, clear domains, a need for differentiated scaling and deployment

Two important clarifications. First: the modular monolith is not the enemy. The Kilometre Zero monolith in 01-06 already had six packages, one per domain, and that modularity was what made the extraction possible; a monolith with well-separated modules and a single database is a legitimate architecture and, for many companies, the best one. Second: the SOA of the 2000s failed not because of the idea of services but because of the ESB (Enterprise Service Bus), which concentrated logic, transformations and orchestration in a central component that ended up being a new monolith, owned by an integration team everybody had to wait for. Kafka, in Kilometre Zero, is deliberately the opposite: a log that neither transforms nor decides anything (02-04); all the intelligence sits in the producers and consumers.

  1. The five defining characteristics

The five properties below are the exam every Kilometre Zero service has to pass. When one of them fails, the system drifts towards the distributed monolith, the worst possible combination: the complexity of the network without the independence that justified it.

Characteristic What it means How Kilometre Zero meets it How it breaks
One service = one business capability The service does something the business recognises by name: "reserve stock", "charge", "assign a delivery" inventory is what an operator calls "the stock"; delivery is what Jordan Hall calls "the fleet" Services per technical tier (database-service, validation-service)
Owns its data Only the service reads and writes its storage; everyone else goes through its API or its events km0_inventory is touched only by inventory; catalog learns the stock through stock.updated (04-05) A JOIN from analytics against the orders tables; a shared products table
Independent deployment A new version can be released without coordinating with anyone or deploying anyone else orders 1.15.0 went out as a canary (07-05) without inventory changing A change in inventory that forces orders to be redeployed the same day
Owning team One team decides, builds, deploys and operates the service (you build it, you run it) The delivery team runs delivery and is on call for it (07-03) A "backend team" that touches all six services
Isolated failures A service that is down or slow degrades its own function, not the platform A slow payments opens the circuit (07-04) and the order stays as "payment pending"; the catalog carries on Chained synchronous calls without timeouts: symptom 2 from 01-06

Note that none of the characteristics mentions size in lines of code. "Micro" is an unfortunate choice of name: the right size is whatever allows the five properties to be met by a team of reasonable size (the informal "two-pizza team" rule). orders is probably the largest service in Kilometre Zero, with its saga orchestrator, its outbox and its Cassandra repository, and that is fine: splitting it into orders-creation and orders-query would add no independence, only network calls.

  1. How to cut the boundaries: lightweight Domain-Driven Design

The hardest question in a microservices architecture is not technical: it is where to cut. Cutting badly produces services that always change together, that call each other in chains for any operation and that share data through the back door. The most useful conceptual tool is Domain-Driven Design (DDD), from which we take just three ideas here, enough to draw boundaries without adopting the whole methodology.

Ubiquitous language

Every domain has a vocabulary that the business and the code must share word for word. A product in the catalog has a name, a description, photos and a price; for inventory it is a SKU with available and reserved units in a market; for delivery it is a parcel with a weight and a refrigeration requirement. It is the same aged-cheese, but with three different meanings, and forcing a single Product class with the fields of all three is what produces the 80-column tables of the monolith. The ubiquitous language is the signal for spotting boundaries: where the same word changes meaning, there is a context boundary.

Bounded contexts

A bounded context is the scope within which a model and its language are valid and coherent. Inside it, a term means one thing; outside, it may mean another. A well-cut microservice coincides with a bounded context (or, sometimes, with a small set of them that one team owns together). The six Kilometre Zero contexts, with the term that names their central model:

Context Central concept (aggregate) Its own terms What does not belong to it
catalog Published product listing, photo, sale price, producer, category, "available" (an indicator, not a quantity) The exact quantity in stock; the price paid in a specific order
orders Order line, status (created, paid, rejected, delivered), amount, saga What is in stock; how the charge is made; where the van is
inventory SKU per market available/reserved units, reservation with expiry, alert threshold The description or the photo; the price
payments Charge gateway, authorization, refund, Idempotency-Key The order lines (only the amount and the reference)
delivery Route and position courier, parcel, time slot, position, assignment The contents of the order beyond weight and refrigeration
analytics Sales fact sales per day/market/producer, forecast, stock alert (read-only) Nothing transactional: it only reads events

Context map

The context map draws the relationships between bounded contexts and, for each one, who is in charge of the contract. The usual relationships are: customer-supplier (the consumer negotiates the contract with the supplier: orders asks inventory for a ReserveStock), conformist (the consumer accepts the supplier's model as is: analytics consumes events exactly as they arrive), anti-corruption layer (the consumer translates the foreign model into its own so as not to be contaminated: payments in front of the external gateway) and published language (an explicit, versioned contract everybody knows: the events on orders.events with their event_id/type/version/timestamp_ms/source/data envelope from 02-04).

flowchart LR
    subgraph Web[Clients]
        BROWSER[Anna's browser]
        APP[Courier app]
        DASH[Producer dashboard]
    end
    BROWSER --> GW[Kong<br/>06-05]
    APP --> GW
    DASH --> GW
    GW --> CAT[catalog<br/>Published product]
    GW --> ORD[orders<br/>Order]
    GW --> DEL[delivery<br/>Route and position]
    ORD -- "gRPC ReserveStock<br/>customer-supplier" --> INV[inventory<br/>SKU per market]
    ORD -- "gRPC Charge<br/>customer-supplier" --> PAY[payments<br/>Charge]
    PAY -- "ACL" --> EXT[(External gateway)]
    ORD -. "order.created / payment.confirmed<br/>published language" .-> K[(Kafka<br/>orders.events)]
    INV -. "stock.updated / stock.reserved" .-> K
    K -. conformist .-> CAT
    K -. conformist .-> DEL
    K -. conformist .-> ANA[analytics<br/>Sales fact]
    DEL -. "delivery.positions" .-> K2[(Kafka<br/>delivery.*)]
    K2 -.-> ANA

The solid arrows are synchronous calls and draw the runtime dependencies: if inventory goes down, orders cannot confirm (though it can still accept, as we saw in 03-05). The dashed arrows are events and create no runtime dependency: catalog keeps working with the latest view it has even if Kafka is down. A map with many solid arrows is the first sign of a questionable cut.

Heuristics for deciding where to cut

When the language alone is not enough to decide, four heuristics help, and it is worth applying all of them because they sometimes contradict each other:

  1. Cohesion: what changes together should live together. Reserving stock and releasing a reservation are the same capability; splitting them into two services would force every change to the rules to be coordinated in two places.
  2. Rate of change: what changes at different rates is best kept apart. payments changes rarely and under audit; catalog changes several times a day during a campaign. Putting them together imposes the payments cadence on the catalog (symptom 3 from 01-06).
  3. Data that changes together in the same transaction: if two entities frequently need to be updated atomically, separating them forces a saga per operation. The available and reserved units of a SKU are always updated together: same service, same row. The order and the stock are updated together only on confirmation: here the saga does pay off, because the rest of the time they are independent.
  4. Teams: a service boundary that runs through a team produces two services that are always deployed together; one that lumps two teams together produces fights over the same code. The boundary must coincide with ownership (section 9).

  1. Cutting mistakes

The three most frequent mistakes have names of their own, and they are worth recognising because all three look like "more microservices" and are the opposite.

Anaemic services or entity services. Cutting by table: products-service, customers-service, orders-service, each a CRUD with no logic. Every business operation ("confirm order") crosses four or five of them in a synchronous chain, and the logic ends up in an "orchestrator service" that is the same old monolith with network latency added. The test: if the service only has GET/POST/PUT/DELETE endpoints on a resource and no business verb (reserve, charge, assign), it is a table with an API, not a capability.

Nanoservices. Cutting so fine that the cost of communication exceeds the cost of the function: one service to compute VAT, another to format addresses. Each one needs deployment, monitoring, a contract, on-call duty. Kilometre Zero decided in 01-06 that orders includes the amount calculation rather than a pricing-service, and in exercise 1 of that lesson that simple coupons live in orders, not in a promotions-service.

Distributed monolith. The result of breaking any of the five characteristics in section 2: services that share a database, that must be deployed together because they share a data model or an internal library with business logic, or that call each other in a synchronous chain for everything. Measurable signs: every release touches more than three repositories; a schema change to one table requires coordinating two teams; the trace of a simple request (07-02) shows more than five synchronous hops.

  1. Data: one database per service and its consequences

The principle "each service owns its data" (01-06, principle 1) is the hardest to accept because it gives up the monolith's most powerful tool: the JOIN. In the monolith, the Montblanc Dairy dashboard was answered with a single query:

-- In the monolith: a single query over a single database
SELECT o.id, o.created_at, o.status, l.product, l.units, p.name,
       d.courier, d.expected_delivery
FROM orders o
JOIN order_lines l ON l.order_id = o.id
JOIN products p    ON p.slug = l.product
LEFT JOIN deliveries d ON d.order_id = o.id
WHERE p.producer = 'montblanc-dairy'
ORDER BY o.created_at DESC;

With orders in Cassandra (04-04), catalog in PostgreSQL and delivery in its own storage, that query no longer exists. There are three ways to rebuild it, and choosing well between them is an architectural decision as important as the cut of the services themselves.

Technique How it works When it fits Cost
API composition One component (the BFF, or the service handling the request itself) calls several services and merges the responses in memory Few calls, small data, a need for immediate freshness Latency = the sum of the calls (or the maximum if they run in parallel); availability is the product of the availabilities; no efficient pagination or cross-filtering
Event-driven materialised view The service that needs the query subscribes to the other services' events and keeps a denormalised copy in its own database, shaped exactly like the query Frequent queries, cross-service filters and sorting, tolerance for a few seconds of lag Eventual consistency; duplicated storage; projection code to maintain
CQRS Formally separating the write model (commands, validations, aggregates) from the read model (views built from the events), even in different stores Very different read and write loads, many different views of the same facts Two models to maintain; reads do not see writes instantly

All three techniques rest on pieces the course already has: composition uses the gRPC clients with timeouts and circuit breakers (02-03, 07-04); materialised views use the idempotent consumer and the outbox from 02-05; and CQRS generalises materialised views.

Eventual consistency as the norm. In 03-05 the saga replaced the global transaction with local steps and compensations; the consequence for queries is that, for a few milliseconds or seconds, order P-2026-000125 exists in orders but not yet in the producer's view. In a microservices architecture that is not a defect to be fixed but the normal state, and the design assumes it: the interface shows "updated 3 s ago", the tests accept the window (07-06), and the only question asked about each piece of data is "how much lag can its reader tolerate?". The table of consistency models per data item from 03-01 is the one revisited here: strong only for available stock and the status of the charge; eventual for everything else.

  1. CQRS: a producer's orders dashboard and the products_view view

The producer dashboard as a CQRS problem

Martha Hill, from Montblanc Dairy, opens her dashboard and expects to see her orders from the last 30 days with product, units, status and expected delivery time, filterable by market and sortable by date. The facts live in three services: the order and its lines in orders, the product name in catalog, the assignment and the expected time in delivery. With API composition it would be necessary to read every order (Cassandra does not filter by producer: its key is the customer, 04-04), query catalog for each product and delivery for each order, and sort in memory: unworkable beyond a few hundred orders.

CQRS solves it by separating the two sides:

  • The command side stays as it is: orders receives "create order", runs the saga, writes to Cassandra and publishes on orders.events through the outbox. It is the model optimised for writing correctly: validations, idempotency, statuses.
  • The query side is a new consumer inside orders (or a query service of its own, if it grows) that subscribes to order.created, payment.confirmed, payment.rejected and the assignment events from delivery, and keeps an orders_by_producer table in PostgreSQL with exactly the columns and the index the dashboard needs. It is the model optimised for reading fast.
flowchart LR
    DASH[Martha's dashboard] -- "1. POST /orders (command)" --> W[orders: write side<br/>saga, outbox]
    W --> C[(Cassandra<br/>km0_orders)]
    W -. "order.created, payment.confirmed" .-> K[(Kafka<br/>orders.events)]
    DEL[delivery] -. "delivery.assigned" .-> K
    K --> P[Projection<br/>idempotent consumer]
    P --> V[(PostgreSQL<br/>orders_by_producer)]
    DASH -- "2. GET /producers/montblanc-dairy/orders (query)" --> R[orders: read side]
    R --> V

The two sides share the name orders and the team, but not the schema or the database: the write side does not know the view exists, and the view can be rebuilt from scratch by replaying the topic from the beginning (the Kafka retention from 02-04 is what makes this possible), which is also how a new column is added to the dashboard without a migration.

Code: the products_view view in catalog

The same pattern, in its simplest form, is what catalog uses to know whether a product is available without querying inventory. In 04-05 the stock.updated event invalidated the Redis cache; here it also feeds a denormalised table in the catalog database, so that the product page and the listings filterable by availability are served with a single local query.

-- km0/sql/catalog/products_view.sql
-- "Hand-made" materialised view: maintained by the consumer, not by PostgreSQL.
CREATE TABLE IF NOT EXISTS products_view (
    slug            TEXT NOT NULL,
    market          TEXT NOT NULL,
    name            TEXT NOT NULL,           -- copied from the catalog model
    producer        TEXT NOT NULL,
    price_cents     INTEGER NOT NULL,
    available       BOOLEAN NOT NULL DEFAULT FALSE,
    approx_units    INTEGER NOT NULL DEFAULT 0,  -- "approx": eventual by design
    stock_version   BIGINT NOT NULL DEFAULT 0,   -- last timestamp_ms applied from inventory
    updated_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (slug, market)
);
CREATE INDEX IF NOT EXISTS idx_products_view_avail ON products_view (market, available, producer);
# km0/services/catalog/stock_projection.py
"""CQRS projection: maintains products_view from stock.updated.

Reuses the idempotent consumer from 02-05 (processed_messages table) and the
Redis invalidation from 04-05. It only adds the write to the view.
"""
import json
import psycopg
from confluent_kafka import Consumer
from services.common import logs, metrics
from services.catalog.cache import invalidate_product   # 04-05

log = logs.get("catalog.projection")
AVAILABLE_THRESHOLD = 1   # a single free unit already counts as "available"


def apply(conn: psycopg.Connection, event: dict) -> None:
    """Applies a stock.updated to the view. Idempotent and tolerant of reordering."""
    d = event["data"]
    with conn.transaction():
        # 1. Deduplicate by event_id (02-05): if it was already applied, leave without touching anything.
        seen = conn.execute(
            "INSERT INTO processed_messages (message_id, consumer) VALUES (%s, 'catalog.projection') ON CONFLICT DO NOTHING RETURNING 1",
            (event["event_id"],)).fetchone()
        if seen is None:
            metrics.counter("km0_projection_duplicates_total").inc()
            return
        # 2. Write only if the event is newer than what has already been applied (01-05: ordering).
        #    If an old event arrives through a late retry, the condition discards it.
        n = conn.execute(
            """
            UPDATE products_view
               SET available = %(avail)s, approx_units = %(units)s,
                   stock_version = %(v)s, updated_at = now()
             WHERE slug = %(slug)s AND market = %(market)s AND stock_version < %(v)s
            """,
            {"slug": d["product"], "market": d["market"],
             "units": d["available"], "avail": d["available"] >= AVAILABLE_THRESHOLD,
             "v": event["timestamp_ms"]}).rowcount
    if n:
        invalidate_product(d["product"], d["market"])   # the 04-05 cache is still valid
    log.info("view updated", product=d["product"], market=d["market"], applied=bool(n))


def loop(conn: psycopg.Connection, consumer: Consumer) -> None:
    consumer.subscribe(["orders.events"])
    while True:
        msg = consumer.poll(1.0)
        if msg is None or msg.error():
            continue
        event = json.loads(msg.value())
        if event["type"] != "stock.updated":
            continue            # this projection ignores the other types
        apply(conn, event)
        consumer.commit(msg)  # ack after writing: at-least-once + idempotency (02-05)

An explanation of the decisions embedded in the code:

  • The products_view row is created by catalog when the producer publishes the product (with available = FALSE); the projection only updates it. If a stock event arrives for a product that has not been published yet, the UPDATE affects no rows and is ignored without error: this is not a problem because, when the product is published, catalog will ask for the initial stock through GetStock (02-03).
  • stock_version < timestamp_ms is the defence against reordering: with 6 partitions on orders.events and key = order id, two events for the same product can land on different partitions and arrive swapped. Without this condition, an old event would overwrite a new one. It is the same reasoning as the CourierTracker from 01-05.
  • The offset commit comes after the transaction: if the process dies halfway, the event is reprocessed and processed_messages discards it.
  • approx_units carries "approx" in its name on purpose: it is a ubiquitous-language decision. Nobody in catalog should use that number to decide whether something can be sold; that is ReserveStock in inventory, with strong consistency.

Code: the bounded context as a package with a public API

The other half of "owns its data" is that the rest of the code cannot skip over the boundary. Inside a Python service, the Kilometre Zero convention is that each bounded context is a package with a single public module, api.py, and everything else private. Anyone importing services.inventory.repository from orders is making the same mistake as a cross-service JOIN.

# km0/services/inventory/api.py
"""Public API of the `inventory` bounded context.

This is the ONLY module in this package that other contexts may import (in the
same process, during the migration) or expose over gRPC (contracts/inventory.proto).
Everything not in here is an implementation detail and may change without notice.
"""
from dataclasses import dataclass
from services.inventory import _domain, _repository, _events   # private: _ prefix

__all__ = ["Reservation", "InsufficientStock", "reserve_stock", "release_reservation", "get_stock"]


@dataclass(frozen=True)
class Reservation:
    reservation_id: str
    order_id: str
    expires_in_s: int


class InsufficientStock(Exception):
    """A business error, not a technical one: the 03-05 saga treats it as a permanent failure."""


def reserve_stock(reservation_id: str, order_id: str, market: str, lines: list[dict]) -> Reservation:
    """Reserves units of several SKUs in a market, atomically.

    Idempotent by reservation_id: repeating the call returns the same Reservation.
    Publishes stock.reserved and stock.updated via the outbox in the same transaction.
    """
    with _repository.transaction() as tx:
        if (existing := _repository.find_reservation(tx, reservation_id)):
            return existing
        for line in lines:
            sku = _repository.lock_sku(tx, line["product"], market)  # SELECT ... FOR UPDATE
            _domain.reserve(sku, line["units"])          # raises InsufficientStock
            _repository.save_sku(tx, sku)
            _events.publish(tx, "stock.updated", {"product": sku.slug, "market": market,
                                                   "available": sku.available})
        reservation = _repository.create_reservation(tx, reservation_id, order_id, lines, expires_in_s=900)
        _events.publish(tx, "stock.reserved", {"order_id": order_id, "reservation_id": reservation_id})
        return reservation


def release_reservation(reservation_id: str) -> None:
    """Saga compensation (C2 in 03-05). Idempotent: releasing twice does not duplicate stock."""
    with _repository.transaction() as tx:
        reservation = _repository.find_reservation(tx, reservation_id)
        if reservation is None or reservation.released:
            return
        for line in reservation.lines:
            sku = _repository.lock_sku(tx, line["product"], reservation.market)
            _domain.release(sku, line["units"])
            _repository.save_sku(tx, sku)
            _events.publish(tx, "stock.updated", {"product": sku.slug, "market": reservation.market,
                                                   "available": sku.available})
        _repository.mark_released(tx, reservation_id)
        _events.publish(tx, "stock.released", {"order_id": reservation.order_id, "reservation_id": reservation_id})


def get_stock(product: str, market: str) -> int:
    """Strongly consistent read (goes to the Patroni primary). The catalog view does
    NOT use this: it uses the event-driven projection."""
    return _repository.read_available(product, market)

The gRPC server from 02-03 (ReserveStock, GetStock) does nothing more than translate protobuf messages into these three functions, and the ServiceInterceptor from 06-04 decides who may call each one. The advantage of having the context as a package with an API is that during the migration (section 8) the monolith could import services.inventory.api in process before the remote service existed, and the switch to gRPC did not touch the callers.

  1. Communication: synchronous, asynchronous, APIs, gateway, BFF and mesh

Synchronous or asynchronous, case by case

Module 2 provided the tools (gRPC in 02-03, Kafka and RabbitMQ in 02-04, asynchronous patterns in 02-05). What belongs to this lesson is the criterion for choosing, which boils down to one question: does the caller need the answer in order to continue, and does it need it now?

Situation Choice Reason Example in Kilometre Zero
The result decides the next step and must be strongly consistent Synchronous (gRPC) The saga cannot move on without knowing whether there is stock orders → inventory.ReserveStock
A person is waiting for the answer on screen Synchronous, with a short timeout and graceful degradation Anna will not wait more than 500 ms (SLO from 07-01) catalog serving the product page; orders accepting the order
Several parties are interested in the same fact Asynchronous (event) The producer must not know its consumers payment.confirmed → delivery, analytics, catalog, the invoices Lambda (08-04)
The receiver may be down and it does not matter Asynchronous Temporal decoupling analytics consuming everything
Long-running or batch work Asynchronous (queue or event) Do not block the caller Generating the invoice; recomputing forecasts
High volume with per-key ordering Asynchronous (Kafka with a key) Per-partition ordering and replay delivery.positions
Notifying an external party that confirms synchronously Synchronous outwards, asynchronous inwards The gateway answers synchronously; the result propagates as an event payments → gateway; then payment.confirmed

Kilometre Zero's rule of thumb, fixed in exercise 2 of 01-06 and confirmed here: synchronous only when the answer governs the next decision; everything else, events. Every synchronous call adds a dependency in availability (the availability of a chain is the product of its links) and in latency (the sum), which is why the context map in section 3 has only two solid arrows between services.

Public API versus internal API

Not all interfaces are equal. A public API (the one consumed by Anna's browser or the courier app, and in the future by third parties) is a long-term commitment: it is versioned, documented (OpenAPI), protected at the gateway, rate limited and kept compatible for months. An internal API (gRPC between orders and inventory) has a few known consumers, changes more freely and has its compatibility verified by the contract tests from 07-06. Confusing the two leads either to exposing internal details (Anna has no interest in the reservation_id) or to bureaucratising internal changes.

API versioning

Every public API changes. The discipline has two levels. Compatible changes (adding an optional field to the response, adding an endpoint, accepting a new parameter with a default value) need no new version: the old client ignores what it does not know. Incompatible changes (renaming or removing a field, changing its type or its semantics, changing error codes) require a new version coexisting with the old one for an announced period. Kilometre Zero versions in the path (/api/v1, /api/v2), which is the most visible option and the one Kong routes without ambiguity (06-05).

# km0/services/orders/http_api.py (excerpt): v1 and v2 served by the same code
from fastapi import FastAPI, Header
from pydantic import BaseModel

app = FastAPI()


class OrderV1(BaseModel):
    id: str
    status: str
    total: float                # in euros, with decimals: the original decision


class OrderV2(BaseModel):
    id: str
    status: str
    total_cents: int            # INCOMPATIBLE change: different type and semantics
    currency: str = "EUR"
    expected_delivery: str | None = None   # NEW optional field: would have been compatible in v1


def _load(order_id: str) -> dict:
    ...  # Cassandra repository (04-04); the internal model stores total_cents


@app.get("/api/v1/orders/{order_id}", response_model=OrderV1, deprecated=True)
def order_v1(order_id: str):
    o = _load(order_id)
    # Adapter: the internal model is already v2; v1 is DERIVED from it, never the other way round.
    return OrderV1(id=o["id"], status=o["status"], total=o["total_cents"] / 100)


@app.get("/api/v2/orders/{order_id}", response_model=OrderV2)
def order_v2(order_id: str):
    o = _load(order_id)
    return OrderV2(id=o["id"], status=o["status"], total_cents=o["total_cents"],
                   expected_delivery=o.get("expected_delivery"))

Three details matter more than the code: the old version is marked deprecated and a retirement date is announced (a Sunset header in the response and a notice to the consumers registered in Kong); who is still using v1 is measured (the km0_http_requests_total{route="/api/v1/..."} metric from 07-01 says when it can be switched off); and the internal model is never kept in two shapes: v1 is a translation of the current model. For events, versioning was already covered in 02-05 (version in the envelope, backward compatibility in the schema).

API gateway and BFF

The gateway from 06-05 is the edge of the public API: routing, JWT, rate limiting, X-Request-Id. Architecturally, its job is for clients to know one domain rather than six services, and for cross-cutting concerns to live in a single place. What it must not do is compose responses: when a client needs data from several services on one screen, the BFF (Backend for Frontend) is the place for that, one per type of client, owned by that client's team. The home screen of the courier app (the day's route, the next parcels, incidents) is the case anticipated in 06-05 and placed in the architecture by the final project (08-05).

Service mesh: when it is worth it

In 06-04 the mesh was presented as a way to get mTLS without code, and in 07-05 as a sidecar that provides timeouts, retries, outlier detection and RED telemetry through configuration; both deferred the decision to this lesson. The criterion is one of number and heterogeneity: with six services, all in Python and sharing services/common/{resilience,metrics,traces}.py, the mesh adds 1-2 ms per hop, memory per sidecar and a control plane to operate, in exchange for little the common library does not already provide. It pays off when services in other languages appear (a recommendations service in Go or Java that cannot use the Python library), when the number of services goes beyond ten and uniform policies become a governance problem, or when security requires mandatory, auditable mTLS (PeerAuthentication: STRICT) without trusting each team to configure it correctly. Kilometre Zero adopts it in the final architecture (08-05) for this last reason, and because the cost of operating it is borne by the internal platform (section 9), not by the product teams.

  1. Migration patterns: assessing the strangler plan

In 01-06 a five-phase strangler fig plan was laid out. Now, with all the pieces built, it is time to assess it phase by phase: what was extracted, with which pattern, and what was learned. First, the four patterns that were used.

  • Strangler fig: a façade is placed in front of the monolith (in Kilometre Zero, Kong), a capability is implemented in a new service and only that route is redirected to it; the monolith loses responsibilities until it is empty. At no point are there two complete systems to maintain.
  • Anti-corruption layer (ACL): a translation layer between the new model and the old one (or an external one), so that the new service's model does not inherit the monolith's defects. It is the "anti-corruption layer" relationship in the context map.
  • Branch by abstraction: inside the monolith, an interface (InventoryPort) is introduced with two implementations, local and remote, selectable through configuration; the remote one is tested with a percentage of the traffic and the local one is removed once validated. It allows migrating without long-lived code branches.
  • Data migration with dual writes and CDC: to move a table from the monolith to the new service's database without downtime: first the history is copied; then the monolith writes to both (dual writes, with the new one as secondary) or, better, a Change Data Capture process (Debezium reading the PostgreSQL WAL, the same WAL as in 03-04) replicates every change to the new store; equality is verified; the direction is reversed (the new service becomes primary and the monolith reads from it); and the old copy is switched off.
Phase (01-06) What was extracted Patterns applied Lessons where it was built What was learned
1 catalog with Redis and km0-photos Strangler fig on the /api/v1/catalog route; data copied cold (reads only, no dual writes) 04-03, 04-05, 06-05 Starting with reads removes risk: rolling back to the monolith meant changing one route in Kong. Symptom 1 disappeared before anything transactional was touched
2 payments isolated ACL in front of the gateway; branch by abstraction (PaymentsPort local/remote); Idempotency-Key 02-05, 07-04 The ACL kept the "charge inside the order transaction" model from being dragged into the new service. The circuit breaker solved symptom 2 even before orders was extracted
3 delivery with Kafka and real time New delivery.positions topic; the positions table stopped being written to PostgreSQL (no history migrated: data that expires) 02-04, 04-04, 05-04, 08-02 Time-series data is not migrated: you stop writing it in the wrong place. This was the phase that introduced the bus, from which everything else later hung
4 orders and inventory Dual writes + CDC for orders (to Cassandra) and for inventory (to km0_inventory); saga replacing the transaction; outbox 03-04, 03-05, 04-04, 02-05 The longest and riskiest phase: two data migrations with equality verification for three weeks. 0.4% of orders with orphan lines were found in the monolith, which the migration cleaned up
5 analytics on Kafka and HDFS Conformist consumer of every topic; /km0/events lake; km0_daily_sales DAG 04-02, 05-02, 05-03, 05-05 Last because it depended on everything already being events. The nightly queries against production disappeared, and symptom 5 with them
Cross-cutting Security, observability, Kubernetes Introduced from phase 1 (Kong, Prometheus, traces) and extended in every phase Modules 6 and 7 Whatever was not put in from the start (traces arrived in phase 2) cost twice as much to add later

One observation about the order: the 01-06 plan put delivery (phase 3) before orders (phase 4) even though symptom 3 (deployments) was the one the teams felt every week. It was the right decision for a reason that was not made explicit at the time: delivery forced Kafka to be set up, and without Kafka phase 4 could not have used the outbox or the materialised views. The order of a migration is dictated by the infrastructure each phase leaves for the next, not only by the pain it relieves.

  1. Organisation: teams, Conway and the internal platform

Conway's law (1967) states that a system mirrors the communication structure of the organisation that builds it. It is an empirical law, but a well-tested one: if two teams have to agree on every change, an interface will appear between their components; if one team owns two services, they will tend to couple because nobody pays the cost of keeping them apart. The practical consequence, sometimes called the inverse Conway manoeuvre, is that the organisation is designed to obtain the desired architecture: in 01-06 Kilometre Zero had four teams (catalog and producers, orders and payments, delivery, data), and the service boundaries followed those lines. orders and payments are two services of the same team, and they are in fact the two that call each other synchronously the most: Conway in action.

Four organisational rules the architecture needs in order to hold up:

  1. Stream-aligned teams, owning a business capability end to end (code, data, deployment, on-call), rather than teams per tier (frontend, backend, database), which turn every feature into a three-team project.
  2. An internal platform team that offers Kubernetes, the mesh, Kafka, observability, the deployment pipeline and the services/common/ libraries as a product with users (the product teams), with its own documentation and SLOs. Without it, every team reinvents the infrastructure, and the uniformity that makes it possible to operate six services is lost. In Kilometre Zero it is the former "data" team, expanded with the people who set up Kubernetes in 07-05.
  3. Contracts as agreements between teams: changing inventory.proto or the stock.updated schema is a negotiation with the consumers, verified by the contract tests from 07-06, not a unilateral decision.
  4. On-call per service, not a central on-call rota: the team that deploys is the one that receives the alert for its SLO (07-01, 07-03). This is what aligns the incentives to build operable services.

  1. When not to use microservices and the return to the modular monolith

Kilometre Zero needed microservices because it had five measurable symptoms and four teams. Most applications have neither, and for them a microservices architecture is a tax with nothing in return: every network boundary costs latency, partial failures, eventual consistency, contracts, deployments and on-call duty.

Signal What it indicates What to do instead
A single team (fewer than 8-10 people) There is no deployment conflict to resolve Modular monolith with packages per domain and an internal API (api.py) per package
The domain is not yet well understood (new product, frequent pivots) The boundaries cut today will be the wrong ones tomorrow, and moving code between services is expensive Modular monolith; cut once the language stabilises
All the traffic fits in two or three replicas of the process No need for differentiated scaling Scale the monolith horizontally; cache and read replicas
Most operations need transactions across several "domains" Every boundary would force a saga Keep together the data that changes together
No capacity to operate Kubernetes, Kafka, distributed observability The platform will consume the team One process, one database, a simple deployment; PaaS
Extremely latency-sensitive (trading, gaming) Every network hop costs In-process modules
The goal is to "modernise" with no concrete symptom Complexity without justification (the 01-06 mistake: "choosing technologies before problems") Write down the symptoms; if there are none, do not migrate

The return to the modular monolith is not a failure, and there are public cases of companies that have regrouped services that called each other in chains for everything. The signals for regrouping two services are the inverse of the signals for cutting: they always change together, all their changes require deploying them at the same time, the trace of any operation always crosses both, or they are owned by the same team and have no differing scaling needs. In Kilometre Zero, if in two years' time payments had no more logic than the adapter to the gateway, it would be a candidate to move back inside orders as a package with its own api.py: the ACL and the idempotency would be kept; the network boundary would not.

  1. Recording decisions: ADR-001

In 01-06 the advice was given to keep an architecture decision log. An ADR (Architecture Decision Record) is a short, numbered document, immutable once accepted (if the decision changes, another ADR is written that supersedes it), with four parts: context, decision, alternatives considered and consequences. Its value lies not in the decision, which is usually well known, but in the context and the consequences, which are what gets forgotten and what, a year from now, will make it possible to tell whether the decision still holds. This is Kilometre Zero's first one, which the final project (08-05) will complete up to ADR-006.

# ADR-001: Extract `inventory` from the monolith as a service with its own database

- **Status**: Accepted (phase 4 of the strangler plan)
- **Date**: 2026-03-10
- **Deciders**: orders and payments team, platform team
- **Supersedes**: —

## Context

Stock lives in the monolith's `products.units` table and is deducted inside the order
confirmation transaction, which also spans the charge at the external payment gateway.
Symptoms (01-06): (2) the gateway's slowness keeps stock locks held for 30 s and
exhausts the PostgreSQL connections; (3) the orders team cannot deploy changes to the
reservation rules without dragging the catalog and delivery along. Stock requires strong
consistency per market (never sell the same item twice), and only `orders` modifies it.
`catalog` only needs to know whether a product is available, with a tolerable lag of
seconds.

## Decision

Extract `inventory` as a bounded context of its own, with its own `km0_inventory` database
(PostgreSQL with Patroni, 03-04/07-03), a gRPC API `contracts/inventory.proto`
(`ReserveStock`, `GetStock`, `WatchChanges`) and the events `stock.updated`,
`stock.reserved`, `stock.released` published via the outbox. The stock reservation becomes
a step of the order saga (03-05), with a 15-minute expiry and the `release_reservation`
compensation. Data migration with CDC (Debezium on the monolith's WAL) and equality
verification for three weeks before reversing the direction.

## Alternatives considered

1. **Leave stock in `orders`** (same database, no boundary). Rejected: it couples the
   reservation rules to the orders deployment cycle and does not allow `catalog` to know
   availability without querying the orders database.
2. **Stock in Cassandra next to the orders**. Rejected: deducting stock needs a
   compare-and-update with strong per-row consistency; Cassandra offers that through
   lightweight transactions (Paxos) at a high cost, and `inventory` does not need its
   write scale (03-02, 04-04).
3. **A 2PC distributed transaction between `orders` and `inventory`**. Rejected: locks
   held while the coordinator decides, and the external gateway does not take part in 2PC (03-05).

## Consequences

- (+) A slow `payments` no longer blocks stock: the reservation expires and the saga compensates.
- (+) The reservation rules are deployed independently.
- (+) `catalog` obtains availability through events (the `products_view` view).
- (−) Eventual consistency between real stock and the catalog: a window of seconds during
  which a sold-out product appears available; accepted and handled with "out of stock" at
  confirmation.
- (−) One more synchronous call on the critical path of confirming an order (p99 +12 ms);
  the 500 ms SLO is still met.
- (−) A new component with on-call duty: Patroni, backups and PITR for `km0_inventory` (07-03).
- (−) The reports that did a `JOIN` of stock with orders are rewritten on top of the event
  lake (phase 5).

ADRs live in the repository (km0/docs/adr/), are reviewed in a pull request like code and are linked from the ticket that originated the change. An ADR that merely says "we use Kafka" with no context and no negative consequences is useless: the negative consequences are the most valuable part.

Common Mistakes and Tips

  • Cutting by table or by technical tier. It produces entity services and synchronous chains. Cut by business capability; if the service has no business verb in its API, rethink it.
  • Sharing the database "just for reads". It is the quietest way to create a distributed monolith: the first ALTER TABLE breaks the neighbour. Event-driven materialised views or API composition; never a cross-service SELECT.
  • Turning every cross-service query into a synchronous call. Availability multiplies and latency adds up. If the query is frequent and tolerates seconds of lag, it is a CQRS projection.
  • Versioning the API by breaking the previous one with no coexistence period. Every new version coexists with the old one, marked deprecated, with a retirement date and a metric that says who is still using it.
  • Migrating in a "big bang". Strangle by route, start with reads, migrate data with CDC and verification, and always keep the way back (a route change in Kong).
  • Ignoring Conway. Service boundaries that do not coincide with teams either couple or fight. Design the organisation together with the architecture.
  • Adopting microservices without symptoms. Write down the symptoms before cutting. If there are none, the modular monolith with an api.py per package is the right answer, and it is reversible.
  • Tip: each bounded context with a single public module; an architecture test (with import-linter or similar) that fails if anyone imports a private module from another context.
  • Tip: write the ADR before the decision, not after: it forces you to enumerate alternatives and negative consequences while you can still change your mind.

Exercises

Exercise 1: cutting a new domain

Kilometre Zero wants to add reviews: Anna rates each product she has received from 1 to 5 and writes a comment; the producer replies; the catalog product page shows the average and the last three comments; the producer dashboard shows the reviews awaiting a reply. A product can only be reviewed from a delivered order. Decide: (a) is it a bounded context of its own, or part of catalog or orders? Justify it with the heuristics from section 3. (b) What communication does it have with the other contexts (synchronous/asynchronous, and in which direction)? (c) How does the catalog product page obtain the average rating without calling the new context synchronously?

Exercise 2: spotting the distributed monolith

A team proposes this implementation of "confirm order": the BFF calls orders, which calls customers (a new service exposing the users table) to validate the address, then pricing (new, computes the amount with VAT) and then inventory, payments and delivery in sequence, all synchronous; analytics queries the orders tables every night with a read-only user; and pricing shares the km0-model library, with the Product and Price classes, with catalog. List every violation of the five characteristics from section 2, name it (entity service, nanoservice, distributed monolith, shared database...) and propose the fix using the techniques from the lesson.

Exercise 3: writing an ADR

Write ADR-002, "Cassandra for orders", with the four parts (context, decision, rejected alternatives, positive and negative consequences), drawing on what was decided in 03-02 and 04-04. Include at least three negative consequences.

Solutions

Exercise 1.

(a) A context of its own, reviews. Language: "review", "average", "producer reply" do not exist in any other context, and "product" here is just an identifier. Cohesion: the rules (one review per delivered line, moderation, a single reply) change together and not with the catalog. Rate of change: it is a new feature that will evolve quickly (moderation, photos, filters), at a pace unlike that of orders. Data that changes together: a review is never written in the same transaction as the order or the product page. Team: it fits with the catalog and producers team, which can own two contexts. Putting it inside catalog would couple comment moderation to catalog deployments during a campaign; inside orders it would pollute the most critical context with logic unrelated to purchasing.

(b) Inbound: reviews consumes order.delivered (a delivery event, asynchronous) to know which lines can be reviewed: it stores (customer, order, product, delivered_at) in its own database; so, when it receives "Anna reviews aged-cheese from P-2026-000124", it checks locally without calling orders. Outbound: it publishes review.created and review.replied on a reviews.events topic. There is no synchronous call between services at all: the only synchronous one is from the BFF/gateway to reviews to create and list.

(c) catalog keeps a projection: it consumes review.created and updates points_sum, review_count and a list of the last three comments (JSON) in products_view (or a products_rating table). The average is computed on read (sum/count) and tolerates seconds of lag. With the idempotent consumer, a reprocessed review is not counted twice. It is exactly the products_view view from section 6 with a different source.

Exercise 2.

Proposed element Violation Name Fix
customers exposes the users table and orders calls it to validate the address A service with no business capability; a synchronous call for stable data Entity service The delivery address travels in the command that creates the order (the customer chose it on screen); the identity already comes in the JWT (06-01). orders keeps a copy of the address in the order (legitimate denormalisation: the order's address does not change if Anna moves house)
pricing computes the amount with VAT in a separate service Network cost for a pure function Nanoservice A function inside orders (the 01-06 decision). If the pricing rules grew (per-market tariffs, complex promotions), it would become a pricing context with its own logic, but consulted through events or with prices published in the catalog
Synchronous chain inventory → payments → delivery delivery does not govern the confirmation; multiplied availability Distributed monolith (temporal coupling) inventory and payments synchronous inside the saga (03-05); delivery consumes payment.confirmed asynchronously and publishes delivery.assigned
analytics with a read-only user on the orders tables Breaks "owns its data"; couples schemas; nightly load on production (symptom 5) Shared database Conformist consumer of orders.events into the lake (05-05)
Shared km0-model library with domain classes A change to Product forces both to be redeployed; denies the ubiquitous language (the same Product for two contexts) Distributed monolith (library coupling) Each context with its own model; what is shared is limited to services/common/ (metrics, logs, traces, resilience: infrastructure, not domain) and to the contracts (.proto, event schemas), which are versioned

Exercise 3. An acceptable ADR-002:

  • Context. Orders grow by 1.2 M a month during a campaign; the access pattern is "a customer's orders by date" and "order by id", with no ad hoc queries; availability is required to accept orders even with a node down or a partition (03-02: orders chooses availability); PostgreSQL with Patroni has a single primary that limits writes and whose failover (30 s) would mean rejecting orders during a campaign.
  • Decision. km0_orders in Cassandra, 3 nodes, RF=3, orders_by_customer and orders_by_id tables (denormalised per query), writes with QUORUM and reads with LOCAL_QUORUM for the order status and ONE for listings; sagas table in the same keyspace.
  • Rejected alternatives. PostgreSQL partitioned by customer with Citus (keeps SQL but introduces a coordinator and distributed transactions the access pattern does not need); MongoDB (documents fit, but the single-primary replication model has the same failover problem); keeping PostgreSQL and accepting the 30 s of unavailability (rejected because of symptom 1: the spikes coincide with when it hurts most).
  • Consequences. (+) Writes scale linearly with nodes; no primary failover; tolerates a node down without rejecting orders. (−) No JOIN and no ad hoc queries: every new question requires a new table or a CQRS view (the producer dashboard from section 6). (−) Tunable consistency that the code must choose on every operation: a wrong level is a silent bug. (−) A new technology to operate: compactions, repairs, snapshots (07-03), team training. (−) Lightweight transactions (Paxos) are expensive: that is why stock does not go here (ADR-001).

Conclusion

A microservices architecture is a system, and not a collection of services, when every service meets five properties: it represents a business capability, it owns its data, it deploys on its own, one team owns it and its failures do not propagate. Boundaries are drawn where the language changes meaning (bounded contexts), guided by cohesion, rate of change, data that changes together and teams, and the Kilometre Zero context map shows the result: only two synchronous arrows (orders → inventory, orders → payments) and everything else through events. Giving up the JOIN forces you to compose in the API, to maintain event-driven materialised views or to formally separate writes from reads with CQRS, as with the Montblanc Dairy dashboard and the products_view table, with eventual consistency as the normal state rather than a defect. Communication is chosen case by case (synchronous only when the answer governs the next decision), public APIs are versioned with coexistence and usage metrics, the gateway and the BFF separate what is cross-cutting from what is specific to each client, and the mesh pays off when heterogeneity or the requirement for mandatory mTLS exceed what a common library can offer. The 01-06 strangler plan has been reviewed phase by phase (strangler fig, ACL, branch by abstraction, CDC), Conway's law explains why the organisation and the architecture must be designed together, the table of signals says when the modular monolith is the right answer, and ADR-001 puts on record why inventory left the monolith and what that costs.

One piece of the platform remains that this map has only named: the dashed delivery.positions arrow ends in Kafka, but Anna does not read Kafka. Between the topic and her screen there is a final stretch with rules of its own (browsers, mobiles, devices on poor networks, thousands of open connections), and that stretch, with the MQTT and WebSockets that 02-01 introduced, is the next lesson: Real-Time Messaging Systems.

Distributed Architectures Course

Module 1: Introduction to Distributed Systems

Module 2: Communication in Distributed Systems

Module 3: Consistency and Replication

Module 4: Distributed Storage

Module 5: Distributed Computing

Module 6: Security in Distributed Systems

Module 7: Monitoring and Maintenance

Module 8: Case Studies and Applications

© Copyright 2026. All rights reserved