The Project | About Us | Contribute | Donations | License

HOME

The previous lesson ended with an awkward question: Kilometre Zero's pipeline builds, signs and deploys version 1.15.0 of orders, runs pytest, verifies contracts and watches the SLOs during the canary, but which test proves that the saga of 03-05 compensates correctly when inventory dies halfway through? How do we know that the circuit breaker of 07-04 opens in time, or that Patroni fails over in under 30 seconds, before it happens on a Saturday afternoon in the middle of Grape Harvest Week? In a program that runs in a single process, a well-written unit test answers almost everything. In a distributed system it does not: non-determinism, partial failures and time mean that many important properties can only be checked by exercising real dependencies, provoking failures on purpose and observing the whole system. This lesson closes Module 7 with the two disciplines that turn confidence into evidence: testing adapted to distributed systems (integration with containers, contracts, load, resilience and testing in production) and chaos engineering, which injects into the platform the very failures we studied in 07-03 and 07-04, with explicit hypotheses and a controlled blast radius.

Why testing a distributed system is harder
The adapted test pyramid
Integration tests with real dependencies: Testcontainers and the saga
Contract tests: consumers, providers and schemas
Load tests: Grape Harvest Week with k6
Resilience tests and testing in production
Deterministic simulation and formal verification: an overview
Chaos engineering: principles, tools and the lifecycle of an experiment
Chaos experiments at Kilometre Zero
Common Mistakes and Tips
Exercises
Conclusion

Why testing a distributed system is harder

A classic test rests on three assumptions: the same input produces the same output, the test environment resembles the real one and, if something fails, it fails completely. Distributed systems break all three.

Difficulty	What it means at Kilometre Zero	Consequence for testing
Non-determinism	Two consumers of the `orders.events` topic may process messages in a different order on every run; the scheduler and the network decide.	A test that passes 99 times and fails once (flaky) may be revealing a real race, not a problem with the test.
Environments	Locally there are no three Patroni nodes, no Kafka partitions, no Kong in front. A mock of `inventory` never returns `UNAVAILABLE` with 800 ms of latency.	Unit tests with test doubles are necessary but insufficient: the behaviour that matters lives in the integration.
Partial failures	`payments` answers, `inventory` does not, and `orders` has to decide what to do with a half-finished `P-2026-000125`.	Intermediate states must be tested (STOCK_RESERVED without payment), not just the happy path and total failure.
Time	Timeouts, retries with backoff, circuit breaker windows, Redis TTLs, `PhiDetector` heartbeats.	The outcome depends on when each thing happens; testing by "waiting 5 seconds" is slow and brittle.
Scale	A bug that shows up at 500 orders per second does not show up at 5.	Some properties can only be verified under realistic load.

From this comes the central idea of the lesson: in a distributed system, the question "does it work?" breaks down into several different questions, each with its own kind of test, and none of them is enough on its own.

The adapted test pyramid

The classic pyramid (many unit tests, some integration tests, few end-to-end tests) still holds, but in distributed systems layers are added that did not exist before and the weight of others changes.

Level	What it verifies	Dependencies	Speed	Recommended amount	Example at Kilometre Zero
Unit	Pure logic of one service	None (doubles)	ms	Many	Computing an order total, `OrderSaga` state transitions with an in-memory repository
Integration with real dependencies	The service against its database, its broker, its cache	Ephemeral containers	seconds	Quite a few	The saga against real PostgreSQL and Kafka (section 3)
Contract	That the provider and consumer of an API or event still agree	Only the contract	ms-s	One per pair	`orders` ↔ `inventory` over `inventory.proto` (section 4)
End-to-end	A complete business flow across all the services	Full environment	minutes	Very few	Creating one of Anna's orders from Kong until `analytics` counts it
Performance and load	That the system meets the SLOs under the expected demand	Production-like environment	minutes-hours	Per release and campaign	Grape Harvest Week with k6 (section 5)
Resilience	That the patterns of 07-04 behave as designed under injected failures	Environment + injection tool	minutes	Per critical pattern	Toxiproxy between `orders` and `inventory` (sections 6 and 9)
In production	That the real version, with real data, behaves	Production	continuous	Always on	The canary of 07-05, the synthetic probes of 07-01

Two practical observations:

End-to-end tests are few and expensive on purpose. Each one needs the whole environment up, takes minutes and, when it fails, points at "something among seven services". They are reserved for the two or three flows that, if broken, sink the business: create an order, pay, deliver.
Contract tests replace many end-to-end ones: if orders and inventory prove separately that they honour the same contract, there is no need to deploy them together to know they will understand each other.

Integration tests with real dependencies: Testcontainers and the saga

A mock of PostgreSQL does not do SELECT ... FOR UPDATE, has no deadlocks and does not reject a transaction for a unique-key violation. A mock of Kafka does not assign partitions or redeliver messages. Testcontainers solves this by starting, from the test itself, ephemeral Docker containers with the real dependencies, and destroying them when it finishes. Every run starts from a clean state and the test runs the same on Martha's laptop and in the pipeline of 07-05.

We are going to test the most delicate part of 03-05: that the order saga, when payments rejects the charge after stock has been reserved, compensates (releases the reservation) and ends in CANCELLED, also leaving the corresponding event in the outbox of 02-05.

# km0/tests/test_order_saga.py
import json
import pytest
import psycopg
from testcontainers.postgres import PostgresContainer
from testcontainers.kafka import KafkaContainer
from kafka import KafkaConsumer

from services.orders.order_saga import OrderSaga, SagaState
from services.orders.repository import SagasRepository
from services.orders.outbox import OutboxPublisher


@pytest.fixture(scope="session")
def postgres():
    # A single PostgreSQL container for the whole test session:
    # starting it costs ~3 s, and each test cleans up its tables.
    with PostgresContainer("postgres:16") as pg:
        yield pg


@pytest.fixture(scope="session")
def kafka():
    with KafkaContainer("confluentinc/cp-kafka:7.6.0") as kf:
        yield kf


@pytest.fixture
def connection(postgres):
    # Real connection; the minimal schema the saga needs is created.
    with psycopg.connect(postgres.get_connection_url().replace("+psycopg2", "")) as con:
        with open("sql/orders_sagas.sql") as f:
            con.execute(f.read())          # tables sagas, outbox, processed_messages
        yield con
        con.execute("TRUNCATE sagas, outbox, processed_messages")
        con.commit()


class FakeInventory:
    """Double of the inventory gRPC client. Records the calls so that
    we can assert that the compensation ran."""
    def __init__(self):
        self.reservations = []
        self.releases = []

    def reserve_stock(self, order_id, product, units):
        self.reservations.append((order_id, product, units))
        return {"reservation_id": f"R-{order_id}"}

    def release_reservation(self, reservation_id):
        self.releases.append(reservation_id)


class RejectingPayments:
    """Payments double that always rejects the charge."""
    def charge(self, order_id, amount):
        raise RuntimeError("card declined")


def test_saga_compensates_when_payments_fails(connection, kafka):
    inventory = FakeInventory()
    outbox = OutboxPublisher(connection, bootstrap=kafka.get_bootstrap_server())
    saga = OrderSaga(
        repo=SagasRepository(connection),
        inventory=inventory,
        payments=RejectingPayments(),
        outbox=outbox,
    )

    saga.run(order_id="P-2026-000125", customer="lucy",
             lines=[("aged-cheese", 2)], amount=18.40)

    # 1. The final state persisted in the `sagas` table is CANCELLED
    row = connection.execute(
        "SELECT state FROM sagas WHERE order_id = %s", ("P-2026-000125",)
    ).fetchone()
    assert row[0] == SagaState.CANCELLED.value

    # 2. The compensation released exactly the reservation that was made
    assert inventory.reservations == [("P-2026-000125", "aged-cheese", 2)]
    assert inventory.releases == ["R-P-2026-000125"]

    # 3. The OrderCancelled event went out through the outbox to a real Kafka
    outbox.publish_pending()
    consumer = KafkaConsumer(
        "orders.events",
        bootstrap_servers=kafka.get_bootstrap_server(),
        auto_offset_reset="earliest",
        consumer_timeout_ms=5000,
    )
    types = [json.loads(m.value)["type"] for m in consumer]
    assert "OrderCancelled" in types

What to understand about this test, line by line:

The postgres and kafka fixtures have scope="session": the containers are started once and shared. Starting Kafka for every test would make the suite unusable.
The connection fixture runs the same sql/orders_sagas.sql that the real deployment uses. If somebody changes the schema and forgets the migration, this test catches it. The final TRUNCATE guarantees isolation between tests.
FakeInventory and RejectingPayments are doubles of the other services, not of the infrastructure. This is the pyramid's criterion: in an integration test of orders, its dependencies (its database, its broker) are used for real, and the other services are replaced, their behaviour being guaranteed separately through contracts.
The three assertions cover the three faces of compensation: persisted state, compensating action executed and event published. If the saga marked CANCELLED but forgot to release the stock, the second assertion would fail, which is exactly the silent bug that would leave Montblanc Dairy with units locked up.
To verify the event, it is consumed from the container's real Kafka. This tests the full Outbox pattern of 02-05, including the serialisation of the event_id/type/version/timestamp_ms/source/data envelope.

The other cases that matter are written with the same structure: inventory returns UNAVAILABLE on the first reservation (it must retry as per 07-04 and not create two reservations), the process dies between STOCK_RESERVED and PAYMENT_CONFIRMED (on restart, the saga must resume from the sagas table), and the same event arrives twice (the processed_messages table must discard the duplicate).

Contract tests: consumers, providers and schemas

When orders (consumer) calls ReserveStock on inventory (provider), both depend on an agreement: fields, types, error codes. A contract test pins that agreement down in an artefact that both sides can verify, without deploying them together.

Consumer-driven contracts (Pact)

In the consumer-driven approach, it is the consumer who declares what it needs. orders writes: "when I call ReserveStock with product=aged-cheese, units=2, I expect a response with a string reservation_id and status=RESERVED". That pact is published to a contract broker and the inventory pipeline verifies it against its real implementation on every change. If inventory renames reservation_id to booking_id, its own pipeline fails before deploying, and the error message says which consumer would break.

Compatibility of protobuf and event schemas

For gRPC and Kafka, much of the contract is already in the schemas of 02-03 (contracts/inventory.proto) and 02-05 (the event envelope). What must be verified is that a schema change is backward compatible: old consumers must keep understanding new messages.

# km0/tests/contract/orders_inventory.py
"""Consumer-provider contract between orders and inventory.

Two checks:
  1. orders declares the fields it uses; inventory must keep serving them.
  2. The new version of inventory.proto is compatible with the published one.
"""
import subprocess
import pytest
from google.protobuf import descriptor_pb2

from contracts import inventory_pb2, inventory_pb2_grpc

# --- 1. Consumer expectations ------------------------------------------
# What `orders` actually uses from the response (grep of the code + review).
FIELDS_USED_BY_ORDERS = {
    "ReserveStockResponse": {"reservation_id", "status", "units_reserved"},
    "GetStockResponse": {"product", "available"},
}

# gRPC codes for which `orders` has retry/compensation logic.
EXPECTED_CODES = {"UNAVAILABLE", "FAILED_PRECONDITION", "DEADLINE_EXCEEDED"}


@pytest.mark.parametrize("message,fields", FIELDS_USED_BY_ORDERS.items())
def test_inventory_still_serves_the_fields_orders_uses(message, fields):
    descriptor = getattr(inventory_pb2, message).DESCRIPTOR
    actual_fields = {f.name for f in descriptor.fields}
    missing = fields - actual_fields
    assert not missing, f"inventory.proto no longer defines {missing} in {message}"


def test_service_exposes_the_contract_methods():
    methods = {m.name for m in inventory_pb2.DESCRIPTOR.services_by_name["Inventory"].methods}
    assert {"ReserveStock", "GetStock"} <= methods


# --- 2. Backward compatibility of the .proto ----------------------------
def _descriptor_set(proto_path: str) -> descriptor_pb2.FileDescriptorSet:
    """Compiles a .proto into a binary FileDescriptorSet with protoc."""
    output = subprocess.check_output([
        "protoc", "--include_imports", "-I", "contracts",
        "--descriptor_set_out=/dev/stdout", proto_path,
    ])
    fds = descriptor_pb2.FileDescriptorSet()
    fds.ParseFromString(output)
    return fds


def _fields_by_message(fds):
    result = {}
    for file in fds.file:
        for msg in file.message_type:
            result[msg.name] = {f.number: (f.name, f.type, f.label) for f in msg.field}
    return result


def test_new_proto_is_compatible_with_published():
    published = _fields_by_message(_descriptor_set("contracts/published/inventory.proto"))
    new = _fields_by_message(_descriptor_set("contracts/inventory.proto"))

    for name, old_fields in published.items():
        assert name in new, f"Message {name} has been removed"
        for number, (field_name, type_, label) in old_fields.items():
            assert number in new[name], (
                f"{name}: field {number} ({field_name}) has been removed; "
                "in protobuf, fields are marked `reserved`, not deleted"
            )
            assert new[name][number][1] == type_, (
                f"{name}.{field_name}: the type of field {number} has changed"
            )

Explanation:

The first part encodes what the consumer needs as data (FIELDS_USED_BY_ORDERS) and checks it against the descriptor of the compiled .proto. It is the lightweight version of the Pact approach: it does not simulate calls, but it catches 90% of breakages (renamed or removed fields, vanished methods).
The second part compares the new .proto with the published copy (the last deployed version, kept in contracts/published/). Protobuf's compatibility rules are concrete: never reuse or delete field numbers, never change types, add fields only as optional. A consumer compiled with the old schema ignores the new fields and keeps working.
For Kafka events, the same idea applies to the envelope of 02-05: the version field exists precisely for this, and an analogous test verifies that the data of version N+1 contains every field of version N. With a schema registry the check can be automatic when the schema is registered.

Load tests: Grape Harvest Week with k6

During Grape Harvest Week, Roble Alto Winery launches the crianza-wine at a discount and orders traffic multiplies eightfold compared with an ordinary Tuesday. The load test answers a very specific question: does version 1.15.0 meet the SLO of 07-01 (99.9% success and p99 < 500 ms) under that traffic? Without thresholds tied to the SLO, a load test only produces pretty graphs.

k6 describes the load in JavaScript and evaluates thresholds when it finishes, returning a non-zero exit code if they are not met, which lets it be integrated as one more stage of the pipeline.

// km0/tests/load/harvest.js
import http from 'k6/http';
import { check, sleep } from 'k6';
import { Rate, Trend } from 'k6/metrics';

// Custom metrics, with names aligned with the Prometheus ones of 07-01
const errors = new Rate('km0_orders_error_rate');
const orderDuration = new Trend('km0_create_order_ms', true);

export const options = {
  // Campaign profile: ramp-up, 20-minute plateau at x8, ramp-down.
  stages: [
    { duration: '3m',  target: 50  },   // 50 virtual users: baseline traffic
    { duration: '5m',  target: 400 },   // climb to the peak (x8)
    { duration: '20m', target: 400 },   // plateau: this is where the real measurement happens
    { duration: '3m',  target: 0   },   // drain
  ],
  thresholds: {
    // Thresholds = SLO of 07-01. If they are not met, k6 exits with an error.
    'km0_orders_error_rate':  ['rate<0.001'],                 // 99.9% success
    'km0_create_order_ms':    ['p(99)<500', 'p(95)<250'],      // p99 < 500 ms
    'http_req_failed':        ['rate<0.001'],
  },
};

const BASE = __ENV.KM0_URL || 'https://staging.km0.example/api/v1';
const TOKEN = __ENV.KM0_TOKEN;   // Keycloak JWT for a test customer

const products = ['crianza-wine', 'crianza-wine', 'crianza-wine', 'aged-cheese', 'pink-tomato'];

export default function () {
  // 1. Catalogue lookup (cheap read, far more frequent than the order)
  const cat = http.get(`${BASE}/catalog/products?market=girona`);
  check(cat, { 'catalog 200': (r) => r.status === 200 });

  // 2. Create order: the operation that governs the SLO
  const product = products[Math.floor(Math.random() * products.length)];
  const body = JSON.stringify({
    customer: `load-${__VU}`,
    lines: [{ product, units: 1 + Math.floor(Math.random() * 3) }],
  });
  const res = http.post(`${BASE}/orders`, body, {
    headers: {
      'Content-Type': 'application/json',
      'Authorization': `Bearer ${TOKEN}`,
      'X-Request-Id': `k6-${__VU}-${__ITER}`,   // Kong propagates it: traceable in Tempo
    },
    tags: { operation: 'create_order' },
  });

  const ok = check(res, {
    'order 201': (r) => r.status === 201,
    'returns id': (r) => r.status === 201 && r.json('order_id') !== undefined,
  });
  errors.add(!ok);
  orderDuration.add(res.timings.duration);

  sleep(1 + Math.random() * 2);   // user "think" time
}

How to read the script:

stages reproduces the shape of the traffic, not just its volume. The peak comes after a ramp, as happens when the campaign newsletter goes out, and there is a 20-minute plateau because memory problems, exhausted connections or Kafka lag appear over time, not in the first minute.
thresholds translates the SLO into executable conditions. That the threshold has the same numbers as alerts.yml of 07-01 is no coincidence: if the test passes in staging but the alert fires in production, the difference is in the environment, not in the criterion.
The X-Request-Id with the k6- prefix makes it possible to filter the test's traces in Loki and Tempo (07-02), and also to exclude them from the business metrics if the test is run against production.
Catalogue reads and order writes are mixed in a realistic proportion. A test that only creates orders overloads km0_orders but leaves Redis and catalog cold, and would not discover that the cache is invalidated too often.
During the plateau, the server-side metrics are watched as well: the HPA of 07-05 must scale orders, the lag of the analytics consumer must not grow without bound and km0_circuit_state must stay closed. The load test is also a test of the automation.

Locust is the Python alternative, with the same philosophy (virtual users with scripted behaviour) and a web interface to follow the test live; the choice between the two is a matter of language preference.

Resilience tests and testing in production

Resilience tests

The patterns of 07-04 have an awkward property: they only act when something goes wrong, so on the happy path they never run. A suite that injects no failures can have 100% line coverage and never once have exercised the opening of a circuit breaker. Resilience tests inject three families of failure:

Injected failure	Typical tool	What should be observed (07-04)
Latency (e.g. +800 ms on `inventory`)	Toxiproxy, `tc netem`	`with_timeout` cuts the wait short; the `RetryBudget` is not exhausted; the p99 rises but does not explode
Errors (5xx, `UNAVAILABLE`)	Service double, Toxiproxy `reset_peer`	`retry` applies backoff with jitter; after N failures the `CircuitBreaker` opens (`km0_circuit_state=2`) and `km0_circuit_rejections_total` grows
Partition (total cut between two services)	Toxiproxy `timeout`, Chaos Mesh `NetworkChaos`	The fallback answers with cached data (`km0_fallback_total`); the saga enters COMPENSATING where appropriate; nothing is left hanging

Section 9 includes one of these tests with code.

Testing in production

However faithful staging is, some things only exist in production: the real data, the real traffic and the real integrations (the payments gateway, the delivery fleet). "Testing in production" does not mean skipping the previous tests, but adding techniques that limit the damage of what is not yet known:

Canary (07-05): the new version receives 5% of the traffic and is automatically compared against the SLO. It is a production test with automatic rollback.
Feature flags: the functionality is deployed switched off and switched on by segment (first for jhall and mhill, then for the Lleida market, then for everyone). It separates deployment from launch and allows switching off without redeploying.
Shadow traffic: Kong duplicates the real requests towards the new version, whose response is discarded but measured. It lets you see how orders 1.16.0 behaves with a Saturday's traffic without any customer suffering it. It demands care with side effects: shadow traffic must not really charge or reserve stock.
Synthetic monitoring (07-01): the blackbox probes continuously run a test purchase flow with a fictitious customer. They are the end-to-end test that never stops running.

Deterministic simulation and formal verification: an overview

There is a family of techniques that attack the non-determinism of section 1 head-on. They are worth knowing even though at Kilometre Zero they are not applied exhaustively:

Jepsen subjects databases and coordination systems (Cassandra, etcd, Kafka, PostgreSQL with Patroni...) to partitions, skewed clocks and crashes, while a checker verifies whether the observed history of operations honours the promised guarantees (linearizability, transaction isolation). Its public reports are the best reading for understanding what the consistency promises of 03-01 really mean, and a good many of the "surprises" they document are what one should expect from the technology one chooses.
TLA+ is a specification language in which the algorithm is described (for instance, the saga of 03-05 with its states and compensations) and a model checker explores all possible interleavings looking for states that violate an invariant ("there is never a COMPLETED order without PAYMENT_CONFIRMED"). It finds design errors that no test would find, because they do not depend on the concrete execution happening to hit the bad case.
Deterministic simulation (the FoundationDB approach): the whole system runs in a single thread with a simulated scheduler and network controlled by a seed. Failures are injected systematically and, when something breaks, the same seed reproduces the failure exactly. It is the most radical answer to the flaky test problem: if it is reproducible, it stops being flaky.

These techniques are expensive and require the system to have been designed for them. For most platforms, Kilometre Zero included, the realistic path is to combine integration and contract tests, load tests with SLO thresholds, and the discipline of the next section.

Chaos engineering: principles, tools and the lifecycle of an experiment

Chaos engineering is the practice of deliberately provoking failures in a system to discover weaknesses before an incident does. It is not "breaking things": it is experimentation with the scientific method. Its principles:

Define the steady state in terms of business metrics and SLOs, not internal resources: "40 orders/min are created with 99.9% success", not "CPU is at 40%".
Formulate a hypothesis about what will happen: "if we kill the Patroni leader, the steady state holds apart from a dip in writes of under 30 s".
Vary real-world events: the failures injected are the ones that actually happen (node crash, latency, partition, full disk, skewed clock), not exotic ones.
Run in production, or as close to it as possible, because it is the only environment whose behaviour matters; but with a controlled blast radius: one pod, one market, 5% of the traffic, with a stop button and during hours when the team is present.
Automate the experiments that have already passed once, so that they keep passing as the platform changes. An experiment that was only run by hand in March says nothing about September's version.

Tools

Tool	Where it acts	What it injects	Use at Kilometre Zero
Chaos Mesh / Litmus	Kubernetes (CRD)	Pod kills, `NetworkChaos` (latency, loss, partition), `IOChaos`, `StressChaos`, `TimeChaos`	Declarative experiments over `k8s/`, schedulable as cron
Toxiproxy	TCP proxy between two services	Latency, jitter, bandwidth limit, cuts, connection reset	Reproducible resilience tests locally and in CI
`tc netem`	Linux kernel (one machine)	Latency, loss, reordering, packet corruption	Kafka/Patroni nodes outside Kubernetes, via Ansible
Custom scripts	Anywhere	`kill -9`, fill the disk (`fallocate`), skew the clock	Cases the tools do not cover

Lifecycle of an experiment

flowchart TD
    A[Define steady state<br/>SLO and business metrics] --> B[Formulate hypothesis<br/>what we expect to happen]
    B --> C[Bound the blast radius<br/>one pod, one market, 5% traffic]
    C --> D[Run the injection<br/>Chaos Mesh / Toxiproxy / tc]
    D --> E{Does the steady<br/>state hold?}
    E -- Yes --> F[Hypothesis confirmed<br/>automate and widen the radius]
    E -- No --> G[Abort: stop the injection<br/>and restore]
    G --> H[Analyse with logs and traces<br/>07-02 · fix the design 07-03/07-04]
    H --> B
    F --> I[Record the result<br/>and the next experiment]

The arrow from G to H is where the value lies: every refuted hypothesis is an incident that will not happen on a Saturday. And the abort button is not optional: before running, you must know exactly how the injection is stopped and how long the system takes to return to the steady state.

Game days

A game day is a planned session (half a day, whole team, with Jordan and Martha on simulated on-call duty) in which several experiments are run back to back and the human side is practised too: did the right alert of 07-01 fire? Was the runbook of 07-03 any use? How long did it take to pinpoint the cause with the traces of 07-02? The outcome is a postmortem with no casualties.

Chaos experiments at Kilometre Zero

The following table is the plan for the platform's first game day. Each row connects a real failure with the mechanism of 07-03 or 07-04 that should absorb it.

Experiment	Injection	Hypothesis	Metrics to watch	Expected outcome thanks to...
Kill the Patroni leader of `km0_inventory`	Chaos Mesh `PodChaos` on the leader pod of the statefulset	Automatic failover; `inventory` writes fail for < 30 s; no reservation is lost or duplicated	`patroni_master`, `inventory` 5xx errors, `km0_stock_units{product}` before/after, `ReserveStock` latency	Patroni + etcd with fencing (07-03); retries with backoff in `orders` (07-04)
Partition between `orders` and `inventory`	Toxiproxy `timeout` / `NetworkChaos` `partition`	Circuit breaker opens in < 10 s; new orders fail fast with a clear message; in-flight sagas compensate; on restore, half-open → closed	`km0_circuit_state`, `km0_circuit_rejections_total`, `km0_fallback_total`, states in the `sagas` table, p99 of `/api/v1/orders`	`CircuitBreaker` and `with_timeout` (07-04); saga COMPENSATING → CANCELLED (03-05)
300 ms latency on Redis	Toxiproxy `latency` on the Redis proxy	The catalogue degrades (p99 rises) but does not fail; the cache is bypassed on timeout and `catalog` is read directly	`km0_request_duration_seconds{service="catalog"}`, cache hit rate, `km0_fallback_total{operation="redis"}`	Fallback and graceful degradation (07-04)
Fill the disk of a Kafka broker	`fallocate` on broker 2's volume via Ansible	The broker drops out of the ISR; producers keep writing to the other two with `acks=all`; no `orders.events` event is lost; the disk alert fires	`kafka_under_replicated_partitions`, `kafka_isr_shrinks_total`, `analytics` lag, `KafkaDiskFull` alert	ISR and replication factor 3 (07-03); alerts of 07-01
Clock skewed +5 min on one `orders` pod	Chaos Mesh `TimeChaos`	Keycloak JWTs are rejected as "not yet valid" on that pod; `/health/ready` takes it out of the load balancing	401 errors per pod, `kube_pod_status_ready`	Probes of 07-03 and 07-05; token validation of 06-01

Partition between `orders` and `inventory` with Toxiproxy

Toxiproxy sits between orders and the gRPC port of inventory; orders is configured to point at the proxy (inventory-proxy:50051) instead of the service directly. From the test, the proxy is manipulated through its HTTP API and the Prometheus metrics that orders exposes on /metrics (07-01) are observed.

# km0/tests/chaos/inventory_partition.py
"""Experiment: partition between orders and inventory.

Hypothesis: with inventory unreachable, the orders circuit breaker opens
in under 10 s, requests fail fast (< 100 ms) and, once the network is
restored, the circuit returns to closed in under 60 s without intervention.
"""
import time
import requests
from toxiproxy import Toxiproxy
from prometheus_client.parser import text_string_to_metric_families

TOXIPROXY = Toxiproxy(server_host="toxiproxy", server_port=8474)
ORDERS_METRICS = "http://orders:8000/metrics"
ORDERS_API = "http://orders:8000/api/v1/orders"
CIRCUIT = "inventory"      # `dependency` label of km0_circuit_state


def metric(name: str, labels: dict) -> float:
    """Reads one specific value from the orders /metrics endpoint."""
    text = requests.get(ORDERS_METRICS, timeout=2).text
    for family in text_string_to_metric_families(text):
        if family.name == name:
            for sample in family.samples:
                if all(sample.labels.get(k) == v for k, v in labels.items()):
                    return sample.value
    raise KeyError(f"{name}{labels} not found")


def wait_until(condition, timeout_s: float, every_s: float = 0.5) -> float:
    """Returns the seconds it took for the condition to hold, or raises AssertionError."""
    start = time.monotonic()
    while time.monotonic() - start < timeout_s:
        if condition():
            return time.monotonic() - start
        time.sleep(every_s)
    raise AssertionError(f"the condition did not hold within {timeout_s} s")


def create_test_order() -> tuple[int, float]:
    start = time.monotonic()
    r = requests.post(ORDERS_API, json={
        "customer": "chaos", "lines": [{"product": "zucchini", "units": 1}]
    }, headers={"X-Request-Id": "chaos-partition"}, timeout=5)
    return r.status_code, (time.monotonic() - start) * 1000


def test_inventory_partition():
    proxy = TOXIPROXY.get_proxy("inventory")

    # --- Steady state: circuit closed and a control order works
    assert metric("km0_circuit_state", {"dependency": CIRCUIT}) == 0
    status, _ = create_test_order()
    assert status == 201

    rejections_before = metric("km0_circuit_rejections_total", {"dependency": CIRCUIT})

    # --- Phase 1: 800 ms latency (above the 300 ms timeout of 07-04)
    proxy.add_toxic(name="slow", type="latency", attributes={"latency": 800, "jitter": 100})
    try:
        for _ in range(5):
            create_test_order()               # they must hit the timeout, not hang
        # --- Phase 2: total cut (partition)
        proxy.add_toxic(name="cut", type="timeout", attributes={"timeout": 0})

        # Hypothesis 1: the circuit opens in < 10 s
        t_open = wait_until(
            lambda: metric("km0_circuit_state", {"dependency": CIRCUIT}) == 2,
            timeout_s=10,
        )
        print(f"circuit opened in {t_open:.1f} s")

        # Hypothesis 2: with the circuit open, failure is fast
        status, ms = create_test_order()
        assert status == 503, "with inventory down an explicit 503 is expected"
        assert ms < 100, f"slow failure ({ms:.0f} ms): the circuit is not short-circuiting"
        assert metric("km0_circuit_rejections_total", {"dependency": CIRCUIT}) > rejections_before
    finally:
        # Abort button: the network is always restored, whatever happens
        proxy.destroy_toxics()

    # Hypothesis 3: automatic recovery (half-open -> closed) in < 60 s
    t_close = wait_until(
        lambda: metric("km0_circuit_state", {"dependency": CIRCUIT}) == 0,
        timeout_s=60,
    )
    print(f"circuit closed again in {t_close:.1f} s")
    status, _ = create_test_order()
    assert status == 201

Key points:

The experiment starts by checking the steady state (circuit closed, a control order works). If the system was already broken before anything was injected, the result would say nothing.
The injection is done in two phases because they are different failures from 07-04: latency exercises with_timeout and the retry budget; the cut exercises the circuit breaker. A clean cut is actually the easy case; "almost acceptable" latency is what exhausts threads.
The hypotheses are numeric and observable from the outside (km0_circuit_state, response time), the same metrics Grafana sees. If the test passes, the dashboard of 07-01 will show exactly that sequence during a real incident.
The finally with destroy_toxics() is the abort button. Without it, a failed assertion would leave the partition active.
The final block verifies what is most often forgotten: that the system recovers on its own. A circuit breaker that opens correctly but needs a restart to close turns every transient failure into an incident.

The same experiment as a Chaos Mesh manifest

Once validated with Toxiproxy, the experiment is declared in Kubernetes so that it can be run periodically against the real environment:

# km0/k8s/chaos/orders-inventory-partition.yaml
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: orders-inventory-partition
  namespace: km0-prod
  labels:
    experiment: "07-06"
spec:
  action: partition          # bidirectional cut; others: delay, loss, duplicate, corrupt
  mode: one                  # blast radius: ONE single orders pod, not all of them
  selector:
    namespaces: [km0]
    labelSelectors:
      app: orders
  direction: both
  target:
    mode: all
    selector:
      namespaces: [km0]
      labelSelectors:
        app: inventory
  duration: "90s"            # restores itself on expiry: automatic abort

mode: one limits the damage to a single orders pod: the rest keep serving, so the global SLO barely suffers even if the hypothesis fails. Widening the radius (mode: fixed-percent at 50%) is the next step, only after the small experiment has passed several times.
duration is the automatic abort: whatever happens, after 90 seconds the partition disappears. For a game day, the pause annotation (experiment.chaos-mesh.org/pause: "true") is added as a manual stop.
The manifest is versioned under k8s/chaos/ and applied from the GitOps pipeline of 07-05 with a Chaos Mesh Schedule, for example every Tuesday at 11:00, during office hours. The hypotheses are checked with the same kind of assertions as the previous script, now run against Prometheus.

Common Mistakes and Tips

Simulating the infrastructure instead of using it. A mock of PostgreSQL or Kafka validates the service's logic against an idea of how the infrastructure behaves, not against the infrastructure. With Testcontainers the cost is a few seconds per session; the benefit is catching the deadlock, the forgotten migration or the badly serialised event.
Treating intermittent tests as noise. A test that fails one time in fifty in a distributed system is usually showing a real race. Retrying it automatically until it passes hides the problem; the right thing is to capture the logs and traces of the failed run (07-02) and reproduce it.
Load thresholds unrelated to the SLO. "The load test passed" means nothing if the threshold was p95 < 2 s and the SLO is p99 < 500 ms. The k6 thresholds and those in alerts.yml must come from the same document.
Testing only the clean cut. A service that is completely down is the easiest failure to handle: it is detected quickly. High latency that stays below the timeout, or errors on 10% of the requests, are what exhaust threads and budgets. Always inject all three families: latency, errors and partition.
Chaos without a hypothesis or an abort button. "Let's kill a node and see what happens" is not an experiment, it is a self-inflicted incident. Without a defined steady state you cannot tell whether the result is good; without an abort, you cannot stop when it is bad.
Running the experiment only once. Every new version of orders or every change in resilience.py can undo what was verified. Experiments that already pass are automated (Chaos Mesh Schedule, pipeline stage) so that they keep passing.
Contracts written by the provider. If inventory documents its API but nobody verifies what orders actually uses, a field "that nobody uses" can be removed and break the saga. The consumer declares the contract and the provider verifies it.
Confusing blast radius with duration. A short experiment on all the pods can do more damage than a long one on a single pod. The radius is bounded first by scope (one pod, one market, a percentage of the traffic) and then by time.

Exercises

Exercise 1: integration test of idempotency

In 02-05 the idempotent consumer of analytics was designed with the processed_messages table. Write, with the same fixture structure as tests/test_order_saga.py, an integration test that publishes the same OrderCompleted event twice (same event_id) on orders.events and verifies that km0_analytics counts a single sale. State what is used for real and what is replaced by a double, and why.

Exercise 2: thresholds and load profile for Artisan Cheese Week

Artisan Cheese Week multiplies the traffic by four (not eight) but concentrates purchases into two hours in the afternoon, with a very abrupt peak. Adapt tests/load/harvest.js: modify stages to represent that profile, adjust the product mix and add a threshold so that the test fails if the HPA of 07-05 has not scaled orders to at least 4 replicas during the plateau (hint: k6 can query an HTTP endpoint; assume GET /internal/replicas exists in staging).

Exercise 3: design a chaos experiment

Design, in the format of the table in section 9, an experiment for the following real failure: the delivery.positions consumer in delivery gets stuck (the process is alive but consumes no messages, for instance because of an internal lock). Define the steady state, the injection (with which tool?), the hypothesis, the metrics to watch and which mechanism from 07-01 to 07-05 should absorb it. Also state the blast radius and the abort button.

Solutions

Solution 1. PostgreSQL (the processed_messages table and the sales table are the core of what is being tested) and Kafka (redelivery is precisely what must be reproduced) are used for real; any external service that analytics calls is replaced by a double (none, in this case). Skeleton:

def test_duplicate_event_is_counted_once(connection, kafka):
    producer = KafkaProducer(bootstrap_servers=kafka.get_bootstrap_server())
    event = envelope(type="OrderCompleted", source="orders",
                     data={"order_id": "P-2026-000126", "amount": 42.0})
    for _ in range(2):                           # same event_id both times
        producer.send("orders.events", json.dumps(event).encode())
    producer.flush()

    consumer = AnalyticsConsumer(connection, bootstrap=kafka.get_bootstrap_server())
    consumer.process_until_empty(timeout_s=5)

    sales = connection.execute("SELECT count(*) FROM sales WHERE order_id = %s",
                               ("P-2026-000126",)).fetchone()[0]
    processed = connection.execute("SELECT count(*) FROM processed_messages WHERE event_id = %s",
                                   (event["event_id"],)).fetchone()[0]
    assert sales == 1 and processed == 1

The assertion on processed_messages checks the mechanism, and the one on sales the business effect; both are needed, because a consumer could insert into processed_messages and still duplicate the sale if it does not do so in the same transaction.

Solution 2. Profile with an abrupt peak and a short plateau, replica threshold checked inside the iteration with a Gauge metric:

import { Gauge } from 'k6/metrics';
const replicas = new Gauge('km0_orders_replicas');

export const options = {
  stages: [
    { duration: '2m',  target: 50  },
    { duration: '1m',  target: 200 },   // abrupt peak: x4 in one minute
    { duration: '10m', target: 200 },
    { duration: '2m',  target: 0   },
  ],
  thresholds: {
    'km0_orders_error_rate':  ['rate<0.001'],
    'km0_create_order_ms':    ['p(99)<500'],
    'km0_orders_replicas':    ['value>=4'],   // evaluated on the last value
  },
};
const products = ['aged-cheese', 'aged-cheese', 'fresh-cheese', 'fresh-cheese', 'pink-tomato'];
// Inside default(), one iteration in every 50:
// if (__ITER % 50 === 0) replicas.add(http.get(`${BASE}/internal/replicas`).json('orders'));

The one-minute ramp is the interesting part: the HPA of 07-05 reacts with a delay (stabilisation window) and the p99 will probably be missed during the first two minutes of the peak. That is not a failure of the test, it is a finding: either orders is pre-warmed before the campaign (a higher minimum replica count that day), or the HPA is tuned.

Solution 3. Steady state: delivery.dashboard receives updated positions from van-3 every 10 s and the lag of the delivery.positions consumer group is < 50 messages. Injection: Chaos Mesh StressChaos is no use (the process is not saturated, it is blocked); the most faithful way is a test feature flag or environment variable that makes the consumer stop calling poll without dying, or a kill -STOP on the process (it freezes it without closing the connection). Radius: a single consumer pod, mode: one. Hypothesis: the pod keeps passing /health/live (the process is alive) but must fail /health/ready within 30 s because the readiness probe checks the age of the last consumed message (07-03); Kubernetes restarts it or takes it out of the group, Kafka reassigns the partition (rebalance) and the lag comes back down; the lag alert of 07-01 fires if it lasts more than two minutes. Metrics: kafka_consumergroup_lag{group="delivery"}, kube_pod_status_ready, age of the last position on delivery.dashboard. Abort: kill -CONT or remove the flag; with Chaos Mesh's duration if PodChaos is used. If the hypothesis fails (the pod stays "ready" while not consuming), the fix belongs to 07-03: the readiness probe must measure progress, not merely that the process answers.

Conclusion

This lesson has completed the answer to the question 07-05 ended with. Confidence that version 1.15.0 works does not come from a single test, but from a pyramid adapted to distributed systems: unit tests for the logic, integration with real dependencies in containers for the saga and the outbox, contracts so that orders and inventory keep understanding each other without being deployed together, load tests with thresholds copied from the SLO for Grape Harvest Week, resilience tests with injected failures so that the patterns of 07-04 run at least once before production, and production techniques (canary, feature flags, shadow traffic, synthetic probes) for what only exists there. On top of all that, chaos engineering turns the failures of 07-03 into experiments with a hypothesis, a bounded blast radius and an abort button, and game days train the people too.

With it, Module 7 closes; its thread has been a single question split into five: how do we know the platform works, and what do we do when it does not.

Layer	Question it answers	Main mechanisms	Lesson
Observability	What is happening now, and is the SLO being met?	Prometheus metrics, burn rate alerts, synthetic probes, Grafana; JSON logs and OpenTelemetry traces with `trace_id`	07-01, 07-02
Recovery	When something goes down, how is it detected and how do we come back?	Health checks, `PhiDetector`, failover with fencing (Patroni, ISR), checkpoints, PITR backups, RPO/RTO, runbooks and postmortems	07-03
Resilience	How do we stop a partial failure from becoming a total one?	Timeouts, retries with a budget, `CircuitBreaker`, `Bulkhead`, fallback and degradation, load shedding	07-04
Automation	How do we deploy and scale without manual intervention or human error?	Ansible, immutable images, Kubernetes with probes and HPA, canary against the SLO, service mesh, GitOps	07-05
Testing and chaos	How do we prove all of the above before it happens?	Testcontainers, contracts, k6 with SLO thresholds, Toxiproxy, Chaos Mesh, game days	07-06

The five layers support one another: load tests are evaluated with the metrics of 07-01, chaos experiments verify the mechanisms of 07-03 and 07-04, and the automation of 07-05 is what allows all of it to run on every release and not just once.

With this module, Kilometre Zero now has all the pieces: communication, consistency, storage, computing, security and operations. So far each lesson has looked at one piece; Module 8 changes scale and looks at the whole platform as an architecture, through case studies. The first question is the one that has been there since Module 1, when the Python+PostgreSQL monolith was split into catalog, orders, inventory, payments, delivery and analytics: what makes a microservices architecture work as a system rather than as a collection of services, where the boundaries are drawn, how data is shared (or not) and what price is paid for each decision. That is where the last module begins.

Testing Distributed Systems and Chaos Engineering

Contents

Why testing a distributed system is harder

The adapted test pyramid

Integration tests with real dependencies: Testcontainers and the saga

Contract tests: consumers, providers and schemas

Consumer-driven contracts (Pact)

Compatibility of protobuf and event schemas

Load tests: Grape Harvest Week with k6

Resilience tests and testing in production

Resilience tests

Testing in production

Deterministic simulation and formal verification: an overview

Chaos engineering: principles, tools and the lifecycle of an experiment

Tools

Lifecycle of an experiment

Game days

Chaos experiments at Kilometre Zero

Partition between `orders` and `inventory` with Toxiproxy

The same experiment as a Chaos Mesh manifest

Common Mistakes and Tips

Exercises

Exercise 1: integration test of idempotency

Exercise 2: thresholds and load profile for Artisan Cheese Week

Exercise 3: design a chaos experiment

Solutions

Conclusion

Distributed Architectures Course

Module 1: Introduction to Distributed Systems

Module 2: Communication in Distributed Systems

Module 3: Consistency and Replication

Module 4: Distributed Storage

Module 5: Distributed Computing

Module 6: Security in Distributed Systems

Module 7: Monitoring and Maintenance

Module 8: Case Studies and Applications

Testing Distributed Systems and Chaos Engineering

Contents

Why testing a distributed system is harder

The adapted test pyramid

Integration tests with real dependencies: Testcontainers and the saga

Contract tests: consumers, providers and schemas

Consumer-driven contracts (Pact)

Compatibility of protobuf and event schemas

Load tests: Grape Harvest Week with k6

Resilience tests and testing in production

Resilience tests

Testing in production

Deterministic simulation and formal verification: an overview

Chaos engineering: principles, tools and the lifecycle of an experiment

Tools

Lifecycle of an experiment

Game days

Chaos experiments at Kilometre Zero

Partition between orders and inventory with Toxiproxy

The same experiment as a Chaos Mesh manifest

Common Mistakes and Tips

Exercises

Exercise 1: integration test of idempotency

Exercise 2: thresholds and load profile for Artisan Cheese Week

Exercise 3: design a chaos experiment

Solutions

Conclusion

Distributed Architectures Course

Module 1: Introduction to Distributed Systems

Module 2: Communication in Distributed Systems

Module 3: Consistency and Replication

Module 4: Distributed Storage

Module 5: Distributed Computing

Module 6: Security in Distributed Systems

Module 7: Monitoring and Maintenance

Module 8: Case Studies and Applications

Partition between `orders` and `inventory` with Toxiproxy