The Project | About Us | Contribute | Donations | License

HOME

The previous lesson ended with a 30-second gap: the time between inv-bcn going quiet and Patroni promoting inv-vlc. During that interval nothing is "broken" as far as the failover machinery is concerned, and yet the whole platform can go down. The reason is that a failure does not stay where it happens: orders waits for inventory, Kong waits for orders, Anna waits for Kong, and every wait consumes a finite resource (a thread, a connection, a slot in a queue). In 01-04 we saw a client with a simple timeout and left the circuit breaker "for later"; in 02-03, gRPC deadlines. This lesson builds the full set of resilience patterns: the techniques that let a caller survive a failing service instead of sinking with it. We implement them from scratch in services/common/resilience.py, apply them to the inventory gRPC client inside orders, reproduce their effect in a simulation, and show where each one should live (library, sidecar, or both).

Anatomy of a cascading failure
Timeouts: per call, overall budget and deadline propagation
Safe retries: backoff, jitter and retry budget
Circuit breaker: the state machine
Fallback and graceful degradation
Bulkhead, backpressure and load shedding
Other patterns: hedged requests, internal rate limiting, idempotency keys
services/common/resilience.py and its use in orders
The simulation: simulations/cascade.py
Where to implement them: library, sidecar (Envoy) or both
Common mistakes and tips
Exercises and solutions
Conclusion

Anatomy of a cascading failure

Take Kilometre Zero's real configuration on a Saturday of Grape Harvest Week: orders has 3 replicas with 32 threads each (96 in total) and receives 100 requests/s; every request calls inventory (ReserveStock, normally 40 ms) and payments. Kong has a pool of 200 connections towards orders.

At 10:12:00, the disk on inv-bcn degrades and ReserveStock starts taking 4 s.

sequenceDiagram
    participant Anna
    participant Kong
    participant O as orders (96 threads)
    participant I as inventory (4 s / call)
    Note over I: 10:12:00 disk degraded
    Anna->>Kong: POST /orders (x100/s)
    Kong->>O: 100 requests/s
    O->>I: ReserveStock (no timeout)
    Note over O: 10:12:01 → 96 threads busy, each waiting 4 s<br/>Real capacity: 96/4 = 24 requests/s. 100 arrive.
    Kong->>O: requests queue up in the socket
    Note over Kong: 10:12:03 → 200 connections exhausted.<br/>Kong answers 503 to EVERYTHING, including GET /catalog
    Anna-->>Anna: "The site is down"
    Note over I: 10:12:30 Patroni fails over to inv-vlc. Too late:<br/>thousands of queued requests and piled-up retries

In numbers: at 4 s per call, 96 threads serve at most 24 requests/s; 100 arrive, so 76 requests pile up every second in the accept queues. Within three seconds Kong has exhausted its pool and starts rejecting every route, including /api/v1/catalog, which does not depend on inventory at all. And when inv-vlc is promoted there is no immediate relief: the queues are full of stale requests whose clients have already given up, and the front end's retries have multiplied the load. This is a cascading failure: one slow component propagates its slowness to everything that waits for it, through the exhaustion of shared resources.

The patterns in this lesson attack each link in the chain:

Link in the cascade	Pattern that breaks it
Waiting 4 s for a call that usually takes 40 ms	Timeout
Retrying without control and multiplying the load	Retries with backoff, jitter and a budget
Still calling something that has been failing for 30 s	Circuit breaker
Having nothing to answer when the call cannot be made	Fallback / degradation
Letting the wait on `inventory` consume the threads `catalog` needs	Bulkhead
Accepting more work than can be done	Backpressure / load shedding
Retrying an operation with side effects (charging twice)	Idempotency keys

Timeouts: per call, overall budget and deadline propagation

A timeout is the decision to stop waiting. Without it, a thread blocked on a socket can stay that way indefinitely; with it, the thread frees the resource and returns an error the rest of the system can handle. Three levels:

Per call: every remote operation has a limit in line with its normal latency. ReserveStock takes 40 ms at p50 and 120 ms at p99: a 300 ms timeout leaves room for queueing and cuts off anything abnormal. The rule of thumb is to set it at 2 to 5 times the observed p99 (07-01), never "30 seconds just in case".
Overall request budget: POST /orders promises p99 < 500 ms. That is the budget; each internal call spends part of it, and the next one receives whatever is left. If ReserveStock has used 280 ms, Charge cannot get a 2 s timeout: it gets 220 ms, or the request is aborted before charging.
Deadline propagation: in gRPC (02-03), the deadline travels in the metadata (grpc-timeout), and every service in the chain inherits it and trims it. When inventory receives a call with a 220 ms deadline, it can hand 200 ms to its SQL query (statement_timeout) and, if that runs out, answer DEADLINE_EXCEEDED instead of doing work nobody will read. HTTP has no standard; Kilometre Zero propagates an X-Deadline header carrying the absolute instant (epoch in ms), which orders turns into a gRPC deadline.

An implementation of the budget:

# km0/services/common/resilience.py (part 1: timeouts)
"""Resilience patterns for service-to-service calls in Kilometre Zero."""
import random
import threading
import time
from dataclasses import dataclass, field
from enum import Enum
from typing import Callable, Iterable, TypeVar

T = TypeVar("T")


class TimedOut(Exception):
    """The request's time budget has been exhausted."""


@dataclass
class TimeBudget:
    """Time budget of a request: an absolute deadline.

    Created once when the request comes in (or from the X-Deadline header)
    and passed to every internal call, which may only use what is left.
    """
    deadline: float   # time.monotonic() by which the request must have finished

    @classmethod
    def from_seconds(cls, seconds: float) -> "TimeBudget":
        return cls(deadline=time.monotonic() + seconds)

    def remaining(self) -> float:
        return max(0.0, self.deadline - time.monotonic())

    def timeout_for(self, maximum: float) -> float:
        """Timeout for one specific call: its own maximum or what is left, whichever is smaller."""
        remaining = self.remaining()
        if remaining <= 0:
            raise TimedOut("request budget exhausted")
        return min(maximum, remaining)


def with_timeout(fn: Callable[[float], T], maximum: float, budget: TimeBudget | None = None) -> T:
    """Runs fn(timeout) with the appropriate timeout.

    fn receives the timeout in seconds and is responsible for applying it (gRPC: timeout=;
    psycopg: statement_timeout; requests: timeout=). No threads are used to "kill"
    the call: the timeout must be honoured by the I/O itself, which is the only reliable place.
    """
    timeout = budget.timeout_for(maximum) if budget else maximum
    return fn(timeout)

Why with_timeout does not wrap the function in a thread with join(timeout) deserves an explanation: in Python (and in almost any runtime) you cannot kill a thread blocked on I/O; an external "timeout" would only hand control back to the caller while leaving the thread blocked all the same, which is precisely the resource we wanted to protect. The timeout that works is the one the networking library applies on the socket.

Safe retries: backoff, jitter and retry budget

Retrying is the natural response to a transient failure, and the easiest way to turn a small problem into a storm. The rules for a retry to be safe:

Only transient errors. UNAVAILABLE, DEADLINE_EXCEEDED, 503, connection refused or reset: yes. INVALID_ARGUMENT, PERMISSION_DENIED, NOT_FOUND, 400, 403: no; the second attempt will fail just the same. FAILED_PRECONDITION ("no stock"): not either; it is an answer, not a failure.
Only idempotent operations (02-05). GetStock: always. ReserveStock with an idempotency key (order_id + line): yes. Charge without a key: never, because a timeout does not tell you whether the charge went through.
Exponential backoff: a growing wait between attempts (100 ms, 200, 400, 800), to give the transient condition time to pass.
Jitter: randomness in the wait. Without it, a thousand clients that failed at the same time retry at the same time, exactly 100 ms later, and then 200 ms later: synchronised waves that keep the service on the floor (thundering herd). With "full jitter" (uniform(0, base × 2^n)), the waves spread out.
Attempt limit (3 in total is a good maximum) and respect for the deadline: a third attempt makes no sense if the budget is already spent.
Retry budget: on top of the per-call limit, a global limit: retries may not exceed, say, 10% of normal calls within a window. If inventory is completely down, 10% extra load is bearable; 200% (3 attempts per request) is what prevents it from recovering.

# km0/services/common/resilience.py (part 2: retries)

class RetryBudget:
    """Retry budget: retries may not exceed a fraction of the calls.

    Simple sliding window: each normal call 'earns' `ratio` credits;
    each retry spends one. No credits, no retry.
    """
    def __init__(self, ratio: float = 0.1, minimum: int = 10):
        self.ratio, self.minimum = ratio, minimum
        self.credits = float(minimum)
        self._lock = threading.Lock()

    def record_call(self) -> None:
        with self._lock:
            self.credits = min(self.credits + self.ratio, 100.0)

    def allow_retry(self) -> bool:
        with self._lock:
            if self.credits >= 1:
                self.credits -= 1
                return True
            return False


def retry(fn: Callable[[float], T], *, max_per_attempt: float, budget: TimeBudget,
          transient: Iterable[type[BaseException]], attempts: int = 3,
          base: float = 0.1, cap: float = 2.0,
          retry_budget: RetryBudget | None = None,
          on_retry: Callable[[int, BaseException, float], None] | None = None) -> T:
    """Runs fn with safe retries.

    - Only retries the exceptions listed in `transient`.
    - Exponential backoff with full jitter: uniform wait in [0, min(cap, base * 2^n)].
    - Never waits longer than what is left of the budget; if nothing is left, propagates the last error.
    - Honours the global retry budget if one is given.
    """
    transient = tuple(transient)
    last: BaseException | None = None
    for n in range(attempts):
        try:
            if retry_budget:
                retry_budget.record_call()
            return with_timeout(fn, max_per_attempt, budget)
        except transient as e:
            last = e
            if n == attempts - 1:
                break
            if retry_budget and not retry_budget.allow_retry():
                break                                      # no credit: do not make the storm worse
            wait = random.uniform(0, min(cap, base * (2 ** n)))
            if wait >= budget.remaining():
                break                                      # we would not make it in time even if we tried
            if on_retry:
                on_retry(n + 1, e, wait)
            time.sleep(wait)
        except TimedOut:
            raise
    assert last is not None
    raise last

Circuit breaker: the state machine

When inventory has been returning UNAVAILABLE to every call for 20 seconds, carrying on is useless for orders (it burns 300 ms of timeout on every request for nothing) and harmful for inventory (which is trying to recover under a hail of requests). The circuit breaker (Nygard, Release It!) is a switch that, past a certain failure ratio, stops making the calls and fails immediately, and every so often lets a probe call through to see whether the target has recovered.

stateDiagram-v2
    [*] --> Closed
    Closed --> Open: failures >= threshold<br/>within the window<br/>(e.g. 50% of 20 calls)
    Open --> HalfOpen: cool-down period<br/>elapses (10 s)
    HalfOpen --> Closed: the probe call(s)<br/>succeed
    HalfOpen --> Open: one probe<br/>call fails
    note right of Closed
        Normal traffic.
        Successes and failures are counted.
    end note
    note right of Open
        Every call fails instantly
        with CircuitOpen. Zero load
        on the dependency.
    end note
    note right of HalfOpen
        Lets N probe calls through;
        the rest keep failing fast.
    end note

Design decisions:

What counts as a failure: transient exceptions and timeouts; not business errors (FAILED_PRECONDITION for lack of stock is a correct answer and must not open the circuit).
Threshold and window: as a ratio with a minimum number of calls (50% failures over at least 20 calls in the last 10 s), not "5 failures in a row", which under heavy traffic is reached by noise alone.
Cool-down: how long to stay open before probing (5-30 s). Too short, and the dependency gets hammered; too long, and the degradation outlives the recovery.
What to return while open: a specific exception (CircuitOpen) that the caller turns into a fallback (section 5), into a 503 with Retry-After, or into a degraded response. Never the original exception in disguise.
One per dependency (and per service instance): orders has one circuit towards inventory and another towards payments; payments failing must not open the inventory one.
Observable: its state is a Prometheus gauge (km0_circuit_state{dependency}, 0/1/2) and every transition is a log entry (circuit_open, the one Jordan found in 07-02).

# km0/services/common/resilience.py (part 3: circuit breaker)
from collections import deque
from prometheus_client import Gauge, Counter

CIRCUIT_STATE = Gauge("km0_circuit_state", "Circuit breaker state: 0 closed, 1 half-open, 2 open",
                      ["service", "dependency"])
CIRCUIT_REJECTIONS = Counter("km0_circuit_rejections_total", "Calls rejected while the circuit was open",
                             ["service", "dependency"])


class State(Enum):
    CLOSED = 0
    HALF_OPEN = 1
    OPEN = 2


class CircuitOpen(Exception):
    def __init__(self, dependency: str, retry_in: float):
        super().__init__(f"circuit open towards {dependency}; retry in {retry_in:.1f}s")
        self.dependency, self.retry_in = dependency, retry_in


class CircuitBreaker:
    def __init__(self, service: str, dependency: str, *, transient: Iterable[type[BaseException]],
                 window_s: float = 10.0, min_calls: int = 20, failure_ratio: float = 0.5,
                 cooldown_s: float = 10.0, half_open_probes: int = 3, log=None):
        self.service, self.dependency = service, dependency
        self.transient = tuple(transient)
        self.window_s, self.minimum, self.ratio = window_s, min_calls, failure_ratio
        self.cooldown, self.probes = cooldown_s, half_open_probes
        self.log = log
        self._state = State.CLOSED
        self._open_since = 0.0
        self._probes_in_flight = 0
        self._results: deque[tuple[float, bool]] = deque()   # (instant, success)
        self._lock = threading.Lock()
        self._publish()

    # --- observable state ----------------------------------------------------
    @property
    def state(self) -> State:
        return self._state

    def _publish(self) -> None:
        CIRCUIT_STATE.labels(self.service, self.dependency).set(self._state.value)

    def _transition(self, new: State) -> None:
        if new is not self._state:
            if self.log:
                self.log.warning(f"circuit_{new.name.lower()}", dependency=self.dependency,
                                 previous=self._state.name.lower())
            self._state = new
            self._publish()

    # --- results window -------------------------------------------------------
    def _prune(self, now: float) -> None:
        while self._results and now - self._results[0][0] > self.window_s:
            self._results.popleft()

    def _failure_ratio(self, now: float) -> tuple[int, float]:
        self._prune(now)
        total = len(self._results)
        failures = sum(1 for _, ok in self._results if not ok)
        return total, (failures / total if total else 0.0)

    # --- decision before calling ----------------------------------------------
    def _before(self) -> None:
        now = time.monotonic()
        with self._lock:
            if self._state is State.OPEN:
                elapsed = now - self._open_since
                if elapsed < self.cooldown:
                    CIRCUIT_REJECTIONS.labels(self.service, self.dependency).inc()
                    raise CircuitOpen(self.dependency, self.cooldown - elapsed)
                self._transition(State.HALF_OPEN)
                self._probes_in_flight = 0
            if self._state is State.HALF_OPEN:
                if self._probes_in_flight >= self.probes:
                    CIRCUIT_REJECTIONS.labels(self.service, self.dependency).inc()
                    raise CircuitOpen(self.dependency, 1.0)
                self._probes_in_flight += 1

    # --- bookkeeping after calling --------------------------------------------
    def _after(self, success: bool) -> None:
        now = time.monotonic()
        with self._lock:
            if self._state is State.HALF_OPEN:
                if success:
                    self._probes_in_flight -= 1
                    if self._probes_in_flight == 0:        # all probes fine: close
                        self._results.clear()
                        self._transition(State.CLOSED)
                else:                                       # one probe failed: reopen
                    self._open_since = now
                    self._transition(State.OPEN)
                return
            self._results.append((now, success))
            total, ratio = self._failure_ratio(now)
            if self._state is State.CLOSED and total >= self.minimum and ratio >= self.ratio:
                self._open_since = now
                self._transition(State.OPEN)

    def run(self, fn: Callable[[], T]) -> T:
        self._before()
        try:
            result = fn()
        except self.transient:
            self._after(success=False)
            raise
        except BaseException:
            self._after(success=True)     # business error: the dependency is working
            raise
        self._after(success=True)
        return result

Fallback and graceful degradation

An open circuit or a timeout does not have to end in an error for Anna. A fallback is the alternative answer when the primary one is unavailable, and graceful degradation is designing the product so that it works "worse, but works":

Situation	Fallback	What it sacrifices
`inventory` is not responding and `catalog` wants to show real-time stock	Show the catalogue with the last stock cached in Redis (04-05) and the label "approximate availability"	Stock accuracy
`inventory` is not responding while creating an order	Accept the order as "pending confirmation": the saga (03-05) stays in an intermediate state and completes when `inventory` comes back; if there is no stock by then, it compensates and tells Anna	Immediate confirmation; the possibility of a later "sorry"
`payments` is not responding	There is no fallback for charging; but there is one for the order: save the intent and charge later (same saga)	Immediacy
`delivery` is not publishing positions	The dashboard shows the last known position with its age	Freshness
Redis down	Go straight to PostgreSQL behind a small bulkhead so as not to knock it over (04-05, stampede)	Latency
`analytics` lagging	Nothing: it is not real time; the DAG catches up	Nothing visible

The fallback is decided per operation and by the business, not in the library: the library raises CircuitOpen; the domain code decides whether that means "approximate stock" or "order pending". And every fallback must be visible: a counter km0_fallback_total{operation} (07-01) and a log entry, because a platform that degrades silently for weeks is a broken platform nobody can see.

Bulkhead, backpressure and load shedding

Bulkhead

A ship's bulkheads stop a leak from flooding the whole hull. In a service, the equivalent is isolating resources per dependency: if orders has 96 threads and inventory hangs, without a bulkhead all 96 end up waiting on inventory; with a bulkhead of 24 permits for inventory, at most 24 threads wait, and the other 72 keep serving whatever does not depend on it (looking up orders, cancelling, health). In Python, a bulkhead is a semaphore acquired without waiting (or with a very short wait):

# km0/services/common/resilience.py (part 4: bulkhead)
BULKHEAD_IN_USE = Gauge("km0_bulkhead_in_use", "Bulkhead permits in use", ["service", "dependency"])


class BulkheadFull(Exception):
    pass


class Bulkhead:
    """Limits how many concurrent calls there can be towards a dependency."""

    def __init__(self, service: str, dependency: str, permits: int, max_wait_s: float = 0.05):
        self.service, self.dependency = service, dependency
        self._sem = threading.Semaphore(permits)
        self.max_wait = max_wait_s

    def run(self, fn: Callable[[], T]) -> T:
        # Wait at most 50 ms for a permit: if there is none, fail fast.
        if not self._sem.acquire(timeout=self.max_wait):
            raise BulkheadFull(f"bulkhead for {self.dependency} is full")
        BULKHEAD_IN_USE.labels(self.service, self.dependency).inc()
        try:
            return fn()
        finally:
            BULKHEAD_IN_USE.labels(self.service, self.dependency).dec()
            self._sem.release()

Connection pools (to PostgreSQL, to gRPC) are natural bulkheads if they are sized per dependency; the usual mistake is a single thread pool for everything.

Backpressure and load shedding

When more work arrives than can be done, there are two possible responses: queue it (and hope the spike passes) or reject it. Queueing without limit is the cascade from section 1: a growing queue is growing latency, until nothing in the queue matters to anyone any more. Backpressure is the consumer telling the producer to slow down (in Kafka, the consumer simply stops calling poll and the producer carries on, because Kafka absorbs it; in gRPC streaming and in TCP, flow control is native; in synchronous HTTP there is no mechanism and all that is left is to reject). Load shedding is rejecting early, with 503 and Retry-After, as soon as the queue exceeds a limit or the waiting time crosses a threshold, before doing any work at all. Rejecting 20% of requests in 1 ms is far better than serving 100% in 8 s, because the remaining 80% get normal service.

And the rejection is done by priority: if orders is saturated, the first requests to be dropped are those from analytics (report queries) and the synthetic probes, and the last is a customer's POST /orders. Kilometre Zero tags requests with X-Priority: high|normal|low at Kong (by route and role) and the load-shedding middleware sheds from lowest to highest:

# km0/services/orders/app.py (excerpt: load shedding)
MAX_IN_FLIGHT = {"high": 90, "normal": 60, "low": 30}   # cumulative limits per priority


@app.middleware("http")
async def load_shedding_middleware(request: Request, call_next):
    priority = request.headers.get("x-priority", "normal")
    in_flight = REQUESTS_IN_FLIGHT.labels(service="orders")._value.get()   # the gauge from 07-01
    if in_flight >= MAX_IN_FLIGHT.get(priority, 60):
        SHED.labels(priority=priority).inc()
        return JSONResponse({"error": "service overloaded"}, status_code=503,
                            headers={"Retry-After": "2"})
    return await call_next(request)

With 96 threads, a low request is rejected as soon as 30 are in flight; a high one only when there are 90. The outcome: on the Saturday of the cascade, Martha's reports fail with 503 and Anna's orders get through.

Other patterns: hedged requests, internal rate limiting, idempotency keys

Hedged requests: for idempotent read operations with a long latency tail, send the same request to a second replica if the first has not answered by the p95, and keep whichever answers first. It lowers the p99 at the cost of about 5% more load. Useful for GetStock against inv-bcn/inv-vlc; never for writes.
Internal rate limiting: the one at the edge (06-05) protects against clients; between services it may also be necessary that analytics does not query inventory more than 50 times/s while the DAG runs. Same algorithm (token bucket), applied in the client or in the sidecar; not developed further here.
Idempotency keys in the public orders API: after a timeout, Anna's front end does not know whether the order was created. With the header Idempotency-Key: <client-generated uuid>, orders stores (key → response) in Redis for 24 h, and if the key comes again it returns the same response without creating another order (02-05). This is what makes retrying from outside safe, and without it no retry of POST /orders by the front end is acceptable.

Summary table:

Pattern	Problem it solves	Typical parameters at Kilometre Zero
Per-call timeout	Waiting indefinitely	2-5 × p99: `ReserveStock` 300 ms, `Charge` 2 s, SQL 200 ms
Budget / propagated deadline	The sum of calls exceeding the SLO	`POST /orders` 500 ms; `X-Deadline` header → gRPC deadline → `statement_timeout`
Retries with backoff + jitter	Transient failures; synchronised storms	3 attempts, base 100 ms, cap 2 s, full jitter, only `UNAVAILABLE`/`DEADLINE_EXCEEDED`, only idempotent operations
Retry budget	Retries that double the load	10% of calls
Circuit breaker	Hammering a dead dependency; burning useless timeouts	50% failures over ≥ 20 calls in 10 s; cool-down 10 s; 3 probes
Fallback / degradation	Having nothing to answer	Cached stock; order "pending confirmation"
Bulkhead	One dependency exhausting the resources of the whole service	24 permits towards `inventory`, 16 towards `payments`, 8 towards Redis
Load shedding	Queues growing without limit	`503` + `Retry-After` above 30/60/90 in flight depending on priority
Hedged requests	Tail latency on reads	Second request at the p95 (80 ms) for `GetStock`
Idempotency key	External retries with side effects	`Idempotency-Key` on `POST /orders`, 24 h in Redis

`services/common/resilience.py` and its use in `orders`

The patterns are composed in an order that matters: from the outside in, bulkhead → circuit breaker → retries → timeout → call. The bulkhead limits how many threads get in; the circuit decides whether it is worth trying at all; the retries repeat the call with a timeout. Putting the retries outside the circuit would mean retrying against an open circuit (pointless); putting the bulkhead inside the retries would acquire and release permits on every attempt (acceptable, but the accounting is worse).

# km0/services/orders/clients/inventory.py
"""Inventory gRPC client used by orders, with the full resilience stack."""
import grpc
from contracts import inventory_pb2, inventory_pb2_grpc
from services.common.logs import log
from services.common.metrics import FALLBACK_TOTAL
from services.common.resilience import (Bulkhead, BulkheadFull, CircuitBreaker, CircuitOpen,
                                        RetryBudget, TimeBudget, TimedOut, retry)


class TransientGrpc(Exception):
    """Wraps the gRPC status codes that are considered transient."""


TRANSIENT_CODES = {grpc.StatusCode.UNAVAILABLE, grpc.StatusCode.DEADLINE_EXCEEDED,
                   grpc.StatusCode.RESOURCE_EXHAUSTED}


class InventoryClient:
    def __init__(self, channel: grpc.Channel):
        self.stub = inventory_pb2_grpc.InventoryStub(channel)
        self.bulkhead = Bulkhead("orders", "inventory", permits=24)
        self.circuit = CircuitBreaker("orders", "inventory", transient=[TransientGrpc], log=log)
        self.retry_budget = RetryBudget(ratio=0.1)

    def _call(self, method, request, timeout: float):
        try:
            return method(request, timeout=timeout)      # the gRPC deadline from 02-03
        except grpc.RpcError as e:
            if e.code() in TRANSIENT_CODES:
                raise TransientGrpc(e.code().name) from e
            raise                                         # NOT_FOUND, FAILED_PRECONDITION...: not transient

    def reserve_stock(self, order_id: str, product: str, units: int, budget: TimeBudget):
        request = inventory_pb2.ReserveStockRequest(order_id=order_id, product=product, units=units,
                                                    idempotency_key=f"{order_id}:{product}")

        def with_retries():
            return retry(
                lambda t: self._call(self.stub.ReserveStock, request, t),
                max_per_attempt=0.3, budget=budget,
                transient=[TransientGrpc], attempts=3, base=0.05, cap=0.2, retry_budget=self.retry_budget,
                on_retry=lambda n, e, w: log.warning("inventory_retry", attempt=n,
                                                     cause=str(e), wait_ms=int(w * 1000)),
            )

        return self.bulkhead.run(lambda: self.circuit.run(with_retries))


# In order_saga.py, the reservation step uses the client and decides the business fallback:
def reserve_step(saga, line, budget):
    try:
        return inventory_client.reserve_stock(saga.order_id, line.product, line.units, budget)
    except (CircuitOpen, BulkheadFull, TimedOut, TransientGrpc) as e:
        # inventory is unavailable: do not fail the order, leave it pending confirmation (03-05)
        FALLBACK_TOTAL.labels(operation="reserve_stock_pending").inc()
        log.warning("reservation_pending_confirmation", order_id=saga.order_id, cause=type(e).__name__)
        saga.mark_pending_confirmation(line)
        return None

Note that FAILED_PRECONDITION ("no stock") is not wrapped in TransientGrpc: it is not retried, it does not open the circuit, and the saga treats it as an ordinary rejection (compensation). The distinction between "the dependency is failing" and "the dependency is saying no" is the most important one in the whole stack.

The simulation: `simulations/cascade.py`

To see the cascade and its mitigation without bringing up the platform, a thread-based simulation: a fake inventory that takes 40 ms and, from second 3 onwards, 4 s; an orders with 32 threads receiving 100 requests/s for 12 s; and a tally of what Anna sees.

# km0/simulations/cascade.py
"""Reproduces a cascading failure and shows the effect of each resilience pattern."""
import threading
import time
from concurrent.futures import ThreadPoolExecutor
from collections import Counter

from services.common.resilience import (Bulkhead, BulkheadFull, CircuitBreaker, CircuitOpen,
                                        TimeBudget, TimedOut, with_timeout)

DURATION_S, RATE, ORDERS_THREADS = 12, 100, 32


class FakeInventory:
    """Takes 40 ms; from 'degrade_at' onwards it takes 4 s (grey failure)."""
    def __init__(self, degrade_at: float):
        self.start, self.degrade_at = time.monotonic(), degrade_at

    def reserve(self, timeout: float):
        latency = 4.0 if time.monotonic() - self.start > self.degrade_at else 0.04
        if latency > timeout:
            time.sleep(timeout)            # the I/O honours the timeout: frees the thread in time
            raise TimeoutError("DEADLINE_EXCEEDED")
        time.sleep(latency)
        return "OK"


def scenario(name: str, call):
    """Generates 100 requests/s for 12 s against a pool of 32 threads and counts the outcomes."""
    results, latencies = Counter(), []
    pool = ThreadPoolExecutor(max_workers=ORDERS_THREADS)
    lock = threading.Lock()

    def request():
        t0 = time.monotonic()
        try:
            call()
            r = "ok"
        except TimeoutError:
            r = "timeout"
        except CircuitOpen:
            r = "circuit_open"
        except BulkheadFull:
            r = "bulkhead_full"
        except TimedOut:
            r = "budget_exhausted"
        with lock:
            results[r] += 1
            latencies.append(time.monotonic() - t0)

    start = time.monotonic()
    sent = 0
    while time.monotonic() - start < DURATION_S:
        pool.submit(request)
        sent += 1
        time.sleep(1 / RATE)
    pool.shutdown(wait=True)
    latencies.sort()
    p50, p99 = latencies[len(latencies) // 2], latencies[int(len(latencies) * 0.99)]
    print(f"{name:<34} sent={sent:4d}  " + "  ".join(f"{k}={v}" for k, v in sorted(results.items()))
          + f"  p50={p50*1000:6.0f}ms p99={p99*1000:6.0f}ms")


if __name__ == "__main__":
    # 1. Nothing at all: every call waits as long as it takes
    inv = FakeInventory(degrade_at=3)
    scenario("no protection", lambda: inv.reserve(timeout=60))

    # 2. 300 ms timeout only
    inv = FakeInventory(degrade_at=3)
    scenario("timeout 300ms", lambda: with_timeout(inv.reserve, 0.3))

    # 3. Timeout + circuit breaker
    inv = FakeInventory(degrade_at=3)
    cb = CircuitBreaker("sim", "inventory", transient=[TimeoutError], min_calls=20,
                        failure_ratio=0.5, cooldown_s=2.0)
    scenario("timeout + circuit breaker", lambda: cb.run(lambda: with_timeout(inv.reserve, 0.3)))

    # 4. Timeout + circuit breaker + bulkhead of 8
    inv = FakeInventory(degrade_at=3)
    cb = CircuitBreaker("sim", "inventory", transient=[TimeoutError], min_calls=20,
                        failure_ratio=0.5, cooldown_s=2.0)
    bh = Bulkhead("sim", "inventory", permits=8)
    scenario("timeout + CB + bulkhead(8)",
             lambda: bh.run(lambda: cb.run(lambda: with_timeout(inv.reserve, 0.3))))

Output (the numbers vary from run to run, the pattern does not):

no protection                      sent=1200  ok=302                                           p50= 4012ms p99=  4088ms
timeout 300ms                      sent=1200  ok=300  timeout=900                                p50=  301ms p99=   312ms
timeout + circuit breaker          sent=1200  circuit_open=838  ok=300  timeout=62               p50=    0ms p99=   301ms
timeout + CB + bulkhead(8)         sent=1200  bulkhead_full=28  circuit_open=850  ok=300  timeout=22  p50=    0ms p99=   118ms

How to read it:

No protection: after second 3, all 32 threads are trapped for 4 s each; the pool only processes 8 requests/s and the other 92 queue up; the program takes over a minute to drain the queue (the real output shows the 4 s p50 of the requests that finished within the time; the rest are still waiting). This is what Kong sees as an exhausted pool.
Timeout: every request occupies a thread for at most 300 ms; the pool sustains 32/0.3 ≈ 106 requests/s, just above the 100 that arrive, so there is no queue, but all of them fail after 300 ms of useless waiting, and inventory keeps receiving 100 calls/s while it tries to recover.
Circuit breaker: after the first 20 failed calls (about 200 ms), the circuit opens; the following ones fail in microseconds (p50 = 0 ms), and every 2 s it lets 3 probes through (timeout=62: the probes that failed). inventory receives 3 calls every 2 s instead of 100/s: it can recover.
Bulkhead: on top of that, there are never more than 8 threads waiting on inventory; the other 24 stay free for everything else (in the simulation there is no "everything else", but the p99 drops to 118 ms because requests no longer wait for a pool thread). The 28 bulkhead_full are the price: requests rejected in 50 ms instead of waiting 300.

What the simulation shows as counters are fallbacks in the real orders: circuit_open and bulkhead_full turn into "order pending confirmation".

Where to implement them: library, sidecar (Envoy) or both

Everything so far lives in the orders code. There is another option: a sidecar proxy (Envoy, the proxy behind the Istio/Linkerd service meshes) next to each service, intercepting outbound traffic and applying timeouts, retries, outlier detection (ejecting a failing instance, a per-instance circuit breaker) and connection limits, without touching the code:

# km0/edge/envoy-orders-sidecar.yaml (excerpt): cluster towards inventory
clusters:
  - name: inventory
    type: STRICT_DNS
    connect_timeout: 0.25s
    http2_protocol_options: {}          # gRPC
    load_assignment:
      cluster_name: inventory
      endpoints:
        - lb_endpoints:
            - endpoint: {address: {socket_address: {address: inventory-1, port_value: 50051}}}
            - endpoint: {address: {socket_address: {address: inventory-2, port_value: 50051}}}
    circuit_breakers:                   # in Envoy, "circuit breaker" = concurrency limits (bulkhead)
      thresholds:
        - priority: DEFAULT
          max_connections: 32
          max_pending_requests: 16
          max_requests: 24              # equivalent to our Bulkhead(permits=24)
          max_retries: 3                # concurrent retries: Envoy's retry budget
    outlier_detection:                  # this is the per-instance circuit breaker
      consecutive_5xx: 5                # (for gRPC, consecutive_gateway_failure)
      interval: 10s
      base_ejection_time: 30s           # instance ejected from load balancing for 30 s
      max_ejection_percent: 50          # never eject all of them

# In the route that sends to that cluster:
routes:
  - match: {prefix: "/km0.inventory.v1.Inventory/"}
    route:
      cluster: inventory
      timeout: 0.3s
      retry_policy:
        retry_on: "unavailable,deadline-exceeded,resource-exhausted"   # transient gRPC codes
        num_retries: 2
        per_try_timeout: 0.3s
        retry_back_off: {base_interval: 0.05s, max_interval: 0.2s}   # with automatic jitter
        retriable_request_headers:
          - name: x-idempotent           # only retry if the client marks the call as idempotent
            exact_match: "true"

Where	Advantages	Drawbacks	What to put there
Library (`resilience.py`, `tenacity`, `pybreaker`, Resilience4j in Java, Polly in .NET)	Knows the business: what is idempotent, which fallback applies, the request budget	Must be implemented (and maintained) in every language; one misconfigured service breaks the policy	Time budget, fallbacks, idempotency, bulkhead per logical dependency, load shedding by priority
Sidecar / mesh (Envoy via Istio or Linkerd, 07-05)	Uniform, no code, per instance (outlier detection sees every replica), observable out of the box	Does not know what is idempotent or which fallback to use; adds latency (1-2 ms) and operational complexity	Per-route timeouts, connection retries, ejection of unhealthy instances, connection limits, mTLS (06-04)
Both	The best of each layer	Risk of duplicated retries	See below

The danger of "both" is retry multiplication: if the library retries 3 times, Envoy retries 3 times for each of those, and Kong another 3, one failed request generates 27 calls to inventory. The rule: retries happen in one layer only (normally the one closest to the business, which knows what is idempotent, or the sidecar if you want uniformity and mark idempotency with a header), and the other layers have num_retries: 0. Timeouts, on the other hand, can and should be in every layer, provided the outer ones are larger than the inner ones (if Kong cuts off at 500 ms and orders at 600, orders keeps working for nobody).

Common Mistakes and Tips

30-second "just in case" timeouts. A long timeout is almost the same as none: the thread is lost anyway. 2-5 × the measured p99, and revisited whenever the latency changes.
Retrying everything. INVALID_ARGUMENT and FAILED_PRECONDITION will fail again; Charge without an idempotency key charges twice. An explicit list of transient exceptions, and only idempotent operations.
Backoff without jitter. A thousand synchronised clients hitting at once every 200 ms. Full jitter, always.
Retries in three layers. Front end × library × sidecar × gateway = storm. One layer retries; the others, zero.
A circuit breaker that counts business errors. "No stock" opens the circuit and every order ends up pending. Only dependency failures, never valid answers.
A threshold based on consecutive failures. At 100 calls/s, 5 in a row is noise. A ratio over a window with a minimum number of calls.
Silent fallback. Weeks showing cached stock without anyone knowing. A counter and a log entry per fallback, and an alert if it persists.
A single thread pool for every dependency. That is the definition of a cascade. A bulkhead (or pool) per dependency.
Queueing without limit "so as not to lose requests". They are lost anyway, only later, and dragging everything down with them. Load shedding with 503 and priority.
Inner timeouts larger than outer ones. The service keeps working for a client that has already left. Shorter at every layer inwards; propagate the deadline.

Exercises

Exercise 1. POST /orders has a 500 ms budget and makes, in order, ReserveStock (300 ms timeout, up to 3 attempts, base 50 ms) and Charge (2 s timeout, no retries). (a) In the worst case, how long can the reservation phase take with retries if every attempt exhausts its timeout, and what does retry do with the second and third attempts given the budget? (b) If the reservation uses 280 ms, with what timeout is Charge called, and what problem does it pose that the gateway takes 318 ms on average (the trace from 07-02)? (c) Propose a design change in the saga that makes the 500 ms SLO compatible with a 300-2,000 ms gateway, and say which pattern from this lesson or from 03-05 you are applying.

Exercise 2. One Saturday, Roble Alto Winery launches an offer and 60% of crianza-wine reservations return FAILED_PRECONDITION (out of stock). Martha notices that km0_circuit_state{dependency="inventory"} goes to 2 and that every order, including those from La Vega Farm, ends up "pending confirmation". (a) What implementation error is there in the client, compared with the code in section 8? (b) While fixing it, Jordan also proposes lowering min_calls to 5 "to react sooner". What risk does that introduce at 100 calls/s? (c) Design an alert (07-01) that warns of a circuit that stays open persistently without warning about brief, healthy openings.

Exercise 3. Kilometre Zero installs Istio (07-05) and by default Envoy applies num_retries: 2 to all gRPC traffic. The resilience.py library still has attempts=3, and the web front end retries POST /orders twice if it gets a timeout. (a) With inventory completely down for 30 s and 100 orders/s, how many calls per second to inventory are generated in the worst case if no layer has a circuit breaker or retry budget? (b) Decide which layer retries and what the others do, justifying it with the table in section 10. (c) What guarantees that the front end's two retries do not create two orders, and what happens if that guarantee does not exist?

Solutions

Exercise 1.

(a) Without a budget, the worst case would be 300 + wait(≤ 50) + 300 + wait(≤ 100) + 300 = up to 1,050 ms. With a TimeBudget of 500 ms: the first attempt uses 300; with_timeout for the second receives min(0.3, 200 ms remaining − wait), that is, roughly 150-200 ms; if that one also runs out, for the third remaining() is ~0 and timeout_for raises TimedOut (or retry detects that wait >= remaining and aborts beforehand). The reservation never exceeds 500 ms; what is sacrificed is the third attempt. (b) timeout_for(2.0) returns min(2.0, 0.22) = 220 ms: the call to the gateway, which takes 318 ms on average, would almost always fail on deadline; and since Charge is not retriable without a key, the order would end in compensation. Worse: the gateway may have charged. (c) Decouple the charge from the response: the saga confirms the reservation, answers 202 "order accepted, payment in progress" and runs Charge asynchronously (next saga step via the outbox, 02-05/03-05) with its own 2 s timeout and idempotency key; if it fails, it compensates the reservation and notifies. This is graceful degradation ("accept the order pending confirmation") combined with the choreographed saga design; the 500 ms SLO now applies to "accept", not to "charge", and a new SLO is needed for "time to payment confirmation" (p99 < 30 s).

Exercise 2.

(a) FAILED_PRECONDITION is being treated as transient: either it was added to TRANSIENT_CODES, or the except grpc.RpcError wraps every code in TransientGrpc. In section 8 only UNAVAILABLE, DEADLINE_EXCEEDED and RESOURCE_EXHAUSTED are wrapped; everything else propagates as is and CircuitBreaker.run counts it as a success of the dependency (except BaseException: self._after(success=True)). With the bug, the 60% of "out of stock" exceeds the 50% ratio and opens the circuit for every producer. (b) At 100 calls/s, 5 calls are 50 ms of traffic: a single lost packet or a replica restarting produces 3 failures out of 5 and opens the circuit on noise, and since the cool-down is 10 s, every false positive costs 10 s of pending orders; the minimum number of calls must be statistically meaningful for the rate (20-50 in 10 s is fine for 100/s; for payments, at 10/s, perhaps 10). (c) avg_over_time(km0_circuit_state{service="orders",dependency="inventory"}[5m]) >= 1.5 with for: 3m, severity page: the 5-minute average of the gauge only reaches 1.5 if it has been open (value 2) for more than three quarters of the time; a 10 s opening followed by a close barely moves the average. Complementary ticket alert: increase(km0_circuit_rejections_total[1h]) > 1000.

Exercise 3.

(a) Per order: front end 3 attempts (1 + 2) × library 3 × Envoy 3 (1 + 2) = 27 calls to inventory per order; at 100 orders/s, 2,700 calls/s against a dead service, and when it comes back, that is the load it will receive in its first second (against the normal 100): "recovering" becomes "going down again". With 300 ms timeouts, moreover, every order takes up to 27 × 300 ms ≈ 8 s to give up. (b) The library (resilience.py) retries: it is the only one that knows ReserveStock is idempotent thanks to idempotency_key and that Charge is not, and the only one that knows the 500 ms budget; Envoy with num_retries: 0 for orders → inventory (or, as a valid alternative, Envoy retries only when the x-idempotent: true header is present and the library does not retry; but not both), keeping outlier_detection (ejecting unhealthy instances, which the library cannot see per instance) and the per-route timeout; the front end does not retry automatically, but shows "we could not confirm, try again" with the same Idempotency-Key, or retries once with a long backoff (2-5 s) and always with the same key. And every layer with a circuit breaker or retry budget: the library with RetryBudget(0.1), Envoy with max_retries under circuit_breakers. (c) The Idempotency-Key header generated by the front end when "buy" is pressed and reused on every retry: orders looks it up in Redis (24 h) and returns the stored response without running the saga again. Without it, a timeout between orders and Kong (the order was created but the response never arrived) followed by a retry creates two orders with two reservations and, if the charge is synchronous, two charges to Anna: the most expensive resilience failure there is.

Conclusion

A distributed system does not fail because one component fails; it fails because the others wait for the failing one until they run dry. The patterns in this lesson break that chain link by link: per-call timeouts sized on the p99, a per-request time budget propagated as a gRPC deadline and statement_timeout; retries only of transient errors and idempotent operations, with exponential backoff, full jitter, an attempt limit, respect for the budget and a retry budget; a circuit breaker with a ratio over a window, a cool-down, probes in half-open, one per dependency and observable in Prometheus; fallbacks decided by the business (approximate stock, order pending confirmation) and always visible; bulkheads so that one dependency cannot exhaust the threads of the whole service; load shedding that rejects early and by priority instead of queueing; and idempotency keys so that retrying from outside is safe. The simulation has shown the cascade in numbers (32 threads, 4 seconds, 8 requests/s of capacity) and how each pattern cuts it, and the Envoy configuration has shown that a good part of this can live in a sidecar, as long as retries live in one layer only.

Up to now, everything has been done by hand: editing docker-compose.yml, starting containers, copying a certificate, running patronictl switchover, tweaking permits=24. Kilometre Zero already has six services, three data stores, Kafka, Redis, MinIO, Kong, Keycloak, Vault, Prometheus, Loki, Tempo, Patroni and etcd: dozens of containers with hundreds of parameters. Nothing built in this module (probes, replicas, sidecars, retry policies) makes sense if every deployment is still a Tuesday afternoon with an ssh session and a checklist in a document, as in the monolith of 01-06. The next lesson covers automation and orchestration: infrastructure as code with Ansible, immutable containers, and Kubernetes as the orchestrator that schedules, autoscales, self-heals and progressively rolls out everything above, with the service mesh applying, without code, the mTLS of 06-04 and today's patterns.

Resilience Patterns: Timeouts, Retries and Circuit Breakers

Contents

Anatomy of a cascading failure

Timeouts: per call, overall budget and deadline propagation

Safe retries: backoff, jitter and retry budget

Circuit breaker: the state machine

Fallback and graceful degradation

Bulkhead, backpressure and load shedding

Bulkhead

Backpressure and load shedding

Other patterns: hedged requests, internal rate limiting, idempotency keys

`services/common/resilience.py` and its use in `orders`

The simulation: `simulations/cascade.py`

Where to implement them: library, sidecar (Envoy) or both

Common Mistakes and Tips

Exercises

Solutions

Conclusion

Distributed Architectures Course

Module 1: Introduction to Distributed Systems

Module 2: Communication in Distributed Systems

Module 3: Consistency and Replication

Module 4: Distributed Storage

Module 5: Distributed Computing

Module 6: Security in Distributed Systems

Module 7: Monitoring and Maintenance

Module 8: Case Studies and Applications

Resilience Patterns: Timeouts, Retries and Circuit Breakers

Contents

Anatomy of a cascading failure

Timeouts: per call, overall budget and deadline propagation

Safe retries: backoff, jitter and retry budget

Circuit breaker: the state machine

Fallback and graceful degradation

Bulkhead, backpressure and load shedding

Bulkhead

Backpressure and load shedding

Other patterns: hedged requests, internal rate limiting, idempotency keys

services/common/resilience.py and its use in orders

The simulation: simulations/cascade.py

Where to implement them: library, sidecar (Envoy) or both

Common Mistakes and Tips

Exercises

Solutions

Conclusion

Distributed Architectures Course

Module 1: Introduction to Distributed Systems

Module 2: Communication in Distributed Systems

Module 3: Consistency and Replication

Module 4: Distributed Storage

Module 5: Distributed Computing

Module 6: Security in Distributed Systems

Module 7: Monitoring and Maintenance

Module 8: Case Studies and Applications

`services/common/resilience.py` and its use in `orders`

The simulation: `simulations/cascade.py`