The Project | About Us | Contribute | Donations | License

HOME

We can now see what happens at TechCorp: logs, metrics and traces. Now it is time for what the course has been promising since 02-01 ("design for failure") and what 03-01, 05-05 and 06-02 kept deferring here: how a service behaves when the network degrades, a dependency stops responding, a message cannot be processed or a saga is left halfway. In a monolith almost every failure is an exception inside one process; in microservices failure is the norm —the eight fallacies from 01-02— and the question that dominates everything is the one raised by the createOrder timeout: "did it run or not?". This lesson turns that question into a set of patterns implemented in Node.js inside @techcorp/common-http and in the RabbitMQ topology.

Types of failure in a distributed system and the question "did it run or not?"
Timeouts: everything that goes over the network has a limit and a budget
Retries with exponential backoff and jitter: what to retry and what not to
Circuit breaker: cutting off before dragging the others down
Bulkhead: isolating dependencies
Fallback and graceful degradation; rate limiting and backpressure
Fail-fast at startup, tolerance at runtime
Recovery in the asynchronous world: retries, TTL queues and DLQ
Stuck sagas: the PAYMENT_TIMEOUT watchdog and reconciliation
Error handling in code: BusinessError, promises and SIGTERM
Resilience testing and basic chaos engineering
Summary table: pattern → problem → where it lives at TechCorp

Types of failure in a distributed system and the question "did it run or not?"

Type of failure	Example at TechCorp	What sets it apart	Appropriate response
Latency	Catalog responds in 1.8 s instead of 40 ms	It works, but slowly; it consumes the caller's resources	Timeout, and watch it (06-01)
Timeout	Customers does not answer within 2 s	We do not know whether it processed	Cancel, and retry only if idempotent
Transient error	`ECONNRESET`, 503 from Catalog during a rolling update, deadlock in PostgreSQL	Goes away by itself in seconds	Retry with backoff
Permanent error	404 nonexistent product, 422 invalid data, bug	Retrying does not help	Fail fast, return RFC 7807
Partial failure	1 of 3 Catalog replicas returns errors	The load balancer spreads traffic, 33% of requests fail	Circuit breaker per instance or `outlierDetection`
Overload	Black Friday: 20× on Catalog, queues growing	Everything is slow; retrying makes it worse	Rate limiting, backpressure, scaling (06-04)
Cascading failure	Catalog slow → Orders exhausts connections → gateway returns 504 for everything	A local failure becomes global	Timeouts + breaker + bulkhead

The timeout is the case that most confuses those coming from the monolith. When createOrder calls Inventory's POST /v1/reservations and the AbortSignal cuts it off at 2 s, there are three possible realities: the request never arrived; it arrived and was processed but the response never came back; it arrived and is still being processed. The caller's code cannot tell them apart. Hence the three tools we already built: idempotency on the receiver (Idempotency-Key, processOnce in 02-05), events with outbox instead of synchronous calls for whatever changes state (the saga), and reconciliation for whatever slips through (section 9). The patterns in this lesson rest on those foundations; without idempotency, none of them is safe.

Timeouts: everything that goes over the network has a limit and a budget

We were already doing it in 03-01: fetch(url, { signal: AbortSignal.timeout(2000) }) with HTTP_TIMEOUT_MS=2000. We elevate it to a rule and add the concept of a budget:

Every operation that goes over the network has a timeout: HTTP, pg (statement_timeout and connectionTimeoutMillis on the pool), MongoDB (serverSelectionTimeoutMS, maxTimeMS), RabbitMQ (publish with confirmation and a time limit).
The timeout of an outgoing call must be smaller than what the caller has left. If the gateway cuts off at 5 s and Orders at 2 s per dependency with two sequential dependencies, Orders has already spent 4 s plus its own work. Rule at TechCorp: gateway 5 s → Orders 3 s per request → 2 s per dependency with the calls in parallel (06-02).
A timeout must be translated into something the caller understands: BusinessError('DEPENDENCY_UNAVAILABLE', …, 503) with Retry-After, not an anonymous 500.

In @techcorp/common-http we centralize the client:

// @techcorp/common-http/src/httpClient.js
const { BusinessError } = require('./errors');

function createHttpClient({ baseUrl, timeoutMs = 2000, name }) {
  async function request(path, { method = 'GET', body, requestId, idempotencyKey } = {}) {
    const headers = { 'content-type': 'application/json', 'x-request-id': requestId };
    if (idempotencyKey) headers['idempotency-key'] = idempotencyKey;
    let res;
    try {
      res = await fetch(baseUrl + path, { method, headers, body: body && JSON.stringify(body), signal: AbortSignal.timeout(timeoutMs) });
    } catch (err) {
      const transient = err.name === 'TimeoutError' || ['ECONNRESET', 'ECONNREFUSED', 'EAI_AGAIN'].includes(err.cause?.code);
      throw new BusinessError('DEPENDENCY_UNAVAILABLE', `${name} is not responding`, 503, { cause: err.name, transient });
    }
    if (res.status === 503 || res.status === 429) throw new BusinessError('DEPENDENCY_UNAVAILABLE', `${name} returned ${res.status}`, 503, { transient: true, retryAfter: res.headers.get('retry-after') });
    if (!res.ok) throw new BusinessError('DEPENDENCY_ERROR', `${name} returned ${res.status}`, 502, { transient: false, status: res.status });
    return res.json();
  }
  return { request };
}

The catch distinguishes network failure/timeout (transient) from anything else; 503/429 from the server are transient too; any other 4xx is permanent and will not be retried.
The transient property on the error is what the following patterns consult: the retry and the breaker do not guess, they ask.
catalogClient and customersClient from 04-04 are rewritten on top of createHttpClient (name: 'catalog', baseUrl: CATALOG_URL).

Retries with exponential backoff and jitter: what to retry and what not to

An immediate retry against a service that is recovering is a small denial of service: a hundred clients retrying at exactly 100 ms produce a synchronized spike. That is why retries use an exponential wait (100, 200, 400 ms…) that is randomized (jitter), and only when (a) the error is transient and (b) the operation is idempotent:

// @techcorp/common-http/src/retry.js
async function retry(fn, { attempts = 3, baseMs = 100, factor = 2, maxMs = 2000, jitter = true, retryIf = (err) => err.transient === true, onRetry } = {}) {
  let lastError;
  for (let attempt = 1; attempt <= attempts; attempt++) {
    try {
      return await fn(attempt);
    } catch (err) {
      lastError = err;
      if (attempt === attempts || !retryIf(err)) throw err;
      let wait = Math.min(maxMs, baseMs * factor ** (attempt - 1));   // 100, 200, 400 …
      if (jitter) wait = Math.random() * wait;                          // "full jitter": between 0 and the wait
      onRetry?.({ attempt, wait, err });
      await new Promise((r) => setTimeout(r, wait));
    }
  }
  throw lastError;
}
module.exports = { retry };

attempts counts the first one: 3 attempts = 1 call + 2 retries.
retryIf by default only accepts errors marked transient; another function can be passed.
Full jitter (random between 0 and the computed wait) is the variant that spreads load best according to the classic AWS study.
onRetry is for logging at warn (06-01) with attempt and wait.

Usage in catalogClient:

getProducts: (ids, { requestId }) =>
  retry(() => http.request(`/v1/products?ids=${ids.join(',')}`, { requestId }),
    { attempts: 3, onRetry: ({ attempt, wait, err }) => logger.warn({ requestId, attempt, wait, err }, 'retrying catalog') })

Mind the budget: 3 attempts × 2 s timeout is up to 6 s, plus the waits. Either the per-attempt timeout is lowered (700 ms) or the attempts are reduced: the sum must fit in the 3 s Orders has. And the decision table to keep in mind:

Situation	Retry?	Why
`GET /v1/products` → 503, 429, `ECONNRESET`, timeout	Yes	Idempotent read, transient error
`GET /v1/customers/{id}` → 404	No	Permanent: the customer does not exist
Any request → 400, 422	No	Our request is wrong; repeating it yields the same
Any request → 500	Carefully	May be transient or a bug; at most 1 retry
`POST /v1/orders` → timeout	No, unless with `Idempotency-Key`	Could create two orders; with the key, the second returns the first (03-01)
`POST /v1/reservations` → timeout	Yes, with `Idempotency-Key` = `orderId`	Inventory already makes it idempotent per order
`INSERT` in PostgreSQL → deadlock (`40P01`)	Yes	Transient by definition; the whole transaction is retried
Publishing to RabbitMQ → channel closed	Yes (the relay does it on the next cycle)	The outbox guarantees nothing is lost

Circuit breaker: cutting off before dragging the others down

If Catalog is down, every POST /v1/orders waits 2 s, retries and waits again: Orders accumulates open requests, exhausts connections and ends up failing too. The circuit breaker observes the failure rate toward a dependency and, when it exceeds a threshold, stops calling it for a while, failing instantly; afterwards it tries one request and, if it goes well, it closes.

stateDiagram-v2
  [*] --> CLOSED
  CLOSED --> OPEN: failures reach the threshold within the window
  OPEN --> HALF_OPEN: openTimeMs elapses
  HALF_OPEN --> CLOSED: the probe request succeeds
  HALF_OPEN --> OPEN: the probe request fails
  CLOSED --> CLOSED: success (resets the counter)

Teaching implementation in the library (in production the opossum library works just as well, with the same semantics and metrics included):

// @techcorp/common-http/src/circuitBreaker.js
const { BusinessError } = require('./errors');

function createCircuitBreaker({ name, failureThreshold = 5, windowMs = 10000, openTimeMs = 30000, isFailure = (err) => err.transient !== false, onChange }) {
  let state = 'CLOSED';
  let failures = [];                     // timestamps of recent failures
  let openUntil = 0;

  function change(next) { if (state !== next) { state = next; onChange?.({ name, state }); } }

  async function execute(fn) {
    if (state === 'OPEN') {
      if (Date.now() < openUntil) throw new BusinessError('DEPENDENCY_UNAVAILABLE', `${name}: circuit open`, 503, { transient: true, retryAfter: Math.ceil((openUntil - Date.now()) / 1000) });
      change('HALF_OPEN');
    }
    try {
      const result = await fn();
      if (state === 'HALF_OPEN') { failures = []; change('CLOSED'); }
      return result;
    } catch (err) {
      if (isFailure(err)) {
        const now = Date.now();
        failures = failures.filter((t) => now - t < windowMs).concat(now);
        if (state === 'HALF_OPEN' || failures.length >= failureThreshold) { openUntil = now + openTimeMs; change('OPEN'); }
      }
      throw err;
    }
  }
  return { execute, state: () => state };
}
module.exports = { createCircuitBreaker };

In CLOSED the failures of the last windowMs are counted; upon reaching failureThreshold (5 in 10 s) it opens for openTimeMs (30 s).
In OPEN it fails without calling, with a 503 and a computed Retry-After: Orders responds in microseconds, not in 2 s, and the gateway can inform the client.
When it expires, the first call goes through in HALF_OPEN: if it goes well, it closes; if it fails, it opens again for another 30 s.
isFailure excludes non-transient errors: a product 404 must not open the circuit.
onChange is the hook for the log (warn) and for the circuit_breaker_state{dependency} gauge (0 closed, 1 half open, 2 open), which 06-05 will be able to watch.

Applied to catalogClient, the order is breaker outside, retry inside: if the circuit is open there is no point in retrying.

const catalogBreaker = createCircuitBreaker({ name: 'catalog', onChange: ({ state }) => { logger.warn({ dependency: 'catalog', state }, 'circuit state changed'); metrics.breaker.set({ dependency: 'catalog' }, STATES[state]); } });

getProducts: (ids, { requestId }) =>
  catalogBreaker.execute(() => retry(() => http.request(`/v1/products?ids=${ids.join(',')}`, { requestId }), { attempts: 2 }))

One breaker per dependency (Catalog, Customers), not a global one; and per process: each Orders replica has its own, which is fine because each one sees its own experience. Relationship with 05-05: Istio's outlierDetection does something similar per destination instance (it ejects the failing pod from load balancing) and without knowing the business; the breaker in code decides per logical dependency and knows what a failure is. They are not mutually exclusive; TechCorp decided to start with the code.

Bulkhead: isolating dependencies

On a ship, bulkheads prevent a leak from flooding the whole hull. In a service, the equivalent is limiting concurrency per dependency so that a slow one does not consume all the resources. Node has no threads to exhaust, but it does have sockets, memory and pool connections:

const pLimit = require('p-limit');
const catalogLimit = pLimit(20);          // at most 20 simultaneous requests to Catalog per replica
const customersLimit = pLimit(20);

getProducts: (ids, opts) => catalogLimit(() => catalogBreaker.execute(() => retry(() => http.request(/* … */), { attempts: 2 })))

If Catalog slows down, at most 20 requests wait; the 21st is queued in memory (or, with p-limit plus a check on activeCount, rejected with an immediate 503). The rest of the service —GET /v1/orders/{id}, the saga consumer— keeps working. Other bulkheads that already exist at TechCorp: separate connection pools for HTTP traffic and for the outbox relay (so it does not run out of pg connections when a thousand requests arrive); the consumer's RabbitMQ channel separate from the publisher's; and, in Kubernetes, resources.limits per pod (05-02) so that one service does not eat the node.

Fallback and graceful degradation; rate limiting and backpressure

When a dependency fails, there are two questions: can I respond with something useful without it? and, if not, how do I fail well?

The BFF from 03-04 shows the list of orders with product names it gets from Catalog. If Catalog is down, the BFF can respond with the orders carrying only the productIds and no names, marking "partial": true. It is a fallback: the user sees their history, if a bit poorer.
Orders cannot degrade the validation of products and prices when creating an order: accepting an order with unknown prices is worse than not accepting it. It responds 503 DEPENDENCY_UNAVAILABLE with Retry-After: 30 (the breaker's) and the RFC 7807 body from 03-01, and the gateway/BFF shows "try again in a few seconds". The order is never left halfway.
Other accepted degradations: Notifications that cannot send email queues it and does not block the saga (the monolith lesson from 01-02); Catalog that cannot reach MongoDB can serve from its cache (06-04) marking Age.

Rate limiting and backpressure are the degradation in the face of overload: better to reject a part with 429 than to degrade 100%. The gateway already limits to 300/min per client (03-04) and responds 429 with Retry-After; services can add their own limit for expensive routes. In RabbitMQ the mechanism is the prefetch(10) from 03-02: the consumer does not accept more than 10 unacknowledged messages; if Inventory is slow, messages stay in the queue instead of piling up in the process's memory, and rabbitmq_queue_messages_ready rises (06-01) so that someone sees it or autoscaling kicks in (06-04). The system pushes back instead of blowing up forward.

Fail-fast at startup, tolerance at runtime

There is a moment when we do not want tolerance: startup. In 04-03 the configuration is validated with zod and the process dies with fatal if ORDERS_DB_URL is missing; Kubernetes retries it and the CrashLoopBackOff is visible. Starting "halfway" and failing on the first request would be worse. On the other hand, once started, a service must not die because a dependency is missing: if RabbitMQ is unavailable at startup, /health/ready returns 503 (the pod receives no traffic) and the service retries the connection with backoff; when it comes back, ready goes to 200. The combination is: invalid config → die; dependency down → not ready, keep trying.

Recovery in the asynchronous world: retries, TTL queues and DLQ

In 03-02 the consumer did a nack with requeue: true once and on the second failure sent to the DLQ. It is simple but has a flaw: the retry is immediate (the message goes back to the head of the queue) and there is no wait. If Payments failed because of a 503 from the PSP, retrying in 5 ms is useless. The idiomatic solution in RabbitMQ is a retry queue with TTL: the failed message is published to a queue with no consumers whose x-message-ttl, upon expiring, returns it to the main queue through a dead-letter exchange.

flowchart LR
  E[techcorp.events] --> Q[orders.saga]
  Q -->|transient failure,<br/>fewer than 5 attempts| R[orders.saga.retry<br/>x-message-ttl 30000]
  R -->|TTL expires → DLX| E2[techcorp.events.retry] --> Q
  Q -->|5 attempts exhausted<br/>or permanent error| D[orders.saga.dlq]

Extension of the library's messaging/topology.js:

async function declareQueueWithRetry(channel, { queue, routingKeys, retryTtlMs = 30000 }) {
  await channel.assertExchange('techcorp.events', 'topic', { durable: true });
  await channel.assertExchange('techcorp.events.dlx', 'topic', { durable: true });
  await channel.assertExchange('techcorp.events.retry', 'direct', { durable: true });

  await channel.assertQueue(queue, { durable: true, deadLetterExchange: 'techcorp.events.dlx' });          // main queue
  await channel.assertQueue(`${queue}.retry`, { durable: true, messageTtl: retryTtlMs,
    deadLetterExchange: 'techcorp.events.retry', deadLetterRoutingKey: queue });                            // waits and comes back
  await channel.assertQueue(`${queue}.dlq`, { durable: true });                                              // final

  for (const rk of routingKeys) await channel.bindQueue(queue, 'techcorp.events', rk);
  await channel.bindQueue(queue, 'techcorp.events.retry', queue);                                            // return from retry
  await channel.bindQueue(`${queue}.dlq`, 'techcorp.events.dlx', '#');
}

orders.saga.retry has no consumer: messages just wait 30 s. When they expire, RabbitMQ resends them through techcorp.events.retry with routing key orders.saga, and that main queue is bound to that exchange with that key: the message reappears 30 s later.
The DLQ receives whatever the main queue rejects with requeue: false.

And the generic consumer decides where each failure goes:

// @techcorp/common-http/src/messaging/consumer.js (excerpt of the handler)
channel.consume(queue, async (msg) => {
  const attempts = (msg.properties.headers['x-attempts'] || 0);
  try {
    await process(msg);
    channel.ack(msg);
  } catch (err) {
    const permanent = err.transient === false || err instanceof BusinessError && !err.transient;
    if (permanent || attempts + 1 >= maxAttempts) {
      logger.error({ eventId: msg.properties.messageId, attempts, err }, permanent ? 'poison message, sending to DLQ' : 'retries exhausted, sending to DLQ');
      channel.nack(msg, false, false);                                                                       // → DLX → <queue>.dlq
    } else {
      logger.warn({ eventId: msg.properties.messageId, attempt: attempts + 1, err }, 'deferred retry');
      channel.publish('', `${queue}.retry`, msg.content, { ...msg.properties, headers: { ...msg.properties.headers, 'x-attempts': attempts + 1 } });
      channel.ack(msg);                                                                                      // already copied to the retry queue
    }
  }
}, { noAck: false });

The x-attempts counter travels in the headers of the message itself (RabbitMQ does not count it for us reliably). The rest of the headers (requestId, traceparent from 06-02) are preserved.
A poison message is one that will always fail: malformed JSON, an orderId that does not exist, a consumer bug with that specific case. Retrying it 5 times every 30 s only delays the others; that is why a permanent error goes to the DLQ on the first try. And that is why prefetch(10) matters: with prefetch(1) a poison message would block the whole queue until the retries ran out.
With maxAttempts = 5 and a 30 s TTL, a message is processed up to 5 times over ~2 minutes: enough for a PSP to recover from a 503; less than the 15 minutes of the reservation's expires_at.

Processing the DLQ. A DLQ with messages is a small incident (in 06-05 it will be an alert to the team that owns the queue). The procedure: (1) inspect in the RabbitMQ console or with the script (headers, x-attempts, the first exception logged with that eventId in Loki); (2) if the cause was transient and has already been resolved (Payments came back), reprocess; (3) if the message is poison, fix the consumer or discard the message, documenting why. The library's scripts/reprocessDlq.js script:

// node scripts/reprocessDlq.js --queue orders.saga --max 50 [--event-filter order.cancelled] [--discard]
async function reprocess({ queue, max, eventFilter, discard }) {
  const channel = await connect(process.env.RABBITMQ_URL);
  let moved = 0;
  while (moved < max) {
    const msg = await channel.get(`${queue}.dlq`, { noAck: false });
    if (!msg) break;
    const type = msg.fields.routingKey;
    if (eventFilter && type !== eventFilter) { channel.nack(msg, false, true); continue; }   // we leave it in the DLQ
    if (discard) { logger.warn({ eventId: msg.properties.messageId, type }, 'discarded from DLQ'); channel.ack(msg); moved++; continue; }
    const headers = { ...msg.properties.headers, 'x-attempts': 0, 'x-reprocess': new Date().toISOString() };
    channel.publish('', queue, msg.content, { ...msg.properties, headers });          // straight to the main queue
    channel.ack(msg);
    logger.info({ eventId: msg.properties.messageId, type }, 'reprocessed from DLQ');
    moved++;
  }
  await channel.close();
}

It runs from an ephemeral pod (kubectl run with the service image) or as a Job; it resets x-attempts and leaves a mark with x-reprocess (and the span with a link from 06-02). It is the most common on-call operation (06-05).

Stuck sagas: the `PAYMENT_TIMEOUT` watchdog and reconciliation

The choreographed saga from 02-05 has no coordinator: if payment.confirmed never arrives (Payments down for longer than the retries last, message in the DLQ), the order stays in STOCK_RESERVED and the reservation blocks stock. In 02-05 we defined the PAYMENT_TIMEOUT watchdog; now we implement it as a periodic job inside orders-service (same process as the relay, with setInterval, and protected with SELECT … FOR UPDATE SKIP LOCKED so that two replicas do not cancel the same order):

// orders-service/src/messaging/sagaWatchdog.js
function createSagaWatchdog({ db, intervalMs = 60000, limitMinutes = 10, logger }) {
  async function cycle() {
    const client = await db.connect();
    try {
      await client.query('BEGIN');
      const { rows } = await client.query(
        `SELECT id FROM orders WHERE status = 'STOCK_RESERVED' AND updated_at < now() - ($1 || ' minutes')::interval
         FOR UPDATE SKIP LOCKED LIMIT 100`, [limitMinutes]);
      for (const { id } of rows) {
        await client.query(`UPDATE orders SET status = 'CANCELLED', cancellation_reason = 'PAYMENT_TIMEOUT', updated_at = now() WHERE id = $1`, [id]);
        await client.query(`INSERT INTO outbox (id, type, payload, headers) VALUES ($1, 'order.cancelled', $2, $3)`,
          [ulid(), { orderId: id, reason: 'PAYMENT_TIMEOUT' }, { source: 'sagaWatchdog' }]);
        logger.warn({ orderId: id }, 'order cancelled due to PAYMENT_TIMEOUT');
      }
      await client.query('COMMIT');
    } catch (err) { await client.query('ROLLBACK'); logger.error({ err }, 'watchdog error'); }
    finally { client.release(); }
  }
  const timer = setInterval(() => cycle().catch(() => {}), intervalMs);
  return { stop: () => clearInterval(timer) };
}

Every minute it looks for orders that have spent more than 10 minutes in STOCK_RESERVED (less than the 15 of expires_at: we cancel before the reservation expires on its own, and the order.cancelled makes Inventory publish stock.released).
Cancellation and event in the same transaction, via outbox: if the watchdog dies halfway, there is no cancelled order without an event.
SKIP LOCKED lets several replicas run the watchdog without stepping on each other.
It is idempotent by construction: on the second cycle the order is no longer in STOCK_RESERVED.

And as a last resort, periodic reconciliation between services: a Kubernetes CronJob (reconcile-reservations, hourly) in Inventory lists its ACTIVE reservations older than 20 minutes, asks Orders (internal GET /v1/orders/{id}) for the status of each one and releases those of CANCELLED or nonexistent orders, logging every discrepancy at error (there should be none: if there are, something is failing in the saga and it must be investigated). Reconciliation does not replace the saga; it is the net under the trapeze.

Error handling in code: `BusinessError`, promises and `SIGTERM`

Rules of TechCorp's code, started in 04-02 and now complete:

BusinessError for the expected, Error for the unexpected. BusinessError('OUT_OF_STOCK', …, 409) is a response; a TypeError is a bug. errorMiddleware translates the former to RFC 7807 with its code and the latter to a generic 500 with requestId (without leaking the stack to the client) and logs it at error.
Do not swallow exceptions. A catch (e) {} turns a visible failure into an order silently lost. If it is caught, something is done: it is translated, retried, compensated or rethrown.
Unhandled promises. A forgotten await in a consumer generates an unhandledRejection that Node 20 turns into a process crash. In server.js: process.on('unhandledRejection', (err) => { logger.fatal({ err }, 'unhandled promise rejection'); process.exit(1); }); better to die loudly and let Kubernetes restart than to carry on in an unknown state. Same with uncaughtException.
Shut down properly on SIGTERM (05-02, terminationGracePeriodSeconds: 30): stop accepting connections (server.close), set /health/ready to 503, wait for in-flight requests to finish, cancel the RabbitMQ consume and wait for already-received messages to be processed or nacked, stop the relay and the watchdog, close pools and flush the tracing SDK (06-02). Without this, every rolling update produces redelivered messages and cut-off requests.

Resilience testing and basic chaos engineering

Patterns that are not tested do not work on the day they are needed. Three levels, from least to most:

Unit tests of the patterns (04-05): retry with a double that fails twice and then responds; the breaker that opens on the fifth failure and closes after the probe; the consumer that sends a 503 to .retry and invalid JSON to .dlq. With jest.useFakeTimers() there is no need to wait a real 30 s.
Injected failures in integration: in Testcontainers, stop the Catalog container mid-test and check that POST /v1/orders returns 503 with Retry-After in under 3 s; add latency with toxiproxy (a proxy that sits between Orders and Catalog and adds 3 s or drops connections) or with tc qdisc add dev eth0 root netem delay 500ms inside the container.
Game days in staging: once a quarter the Platform team kills pods at random (kubectl delete pod -l app=payments-service), cuts RabbitMQ for 5 minutes, fills the DLQ or slows Catalog to 2 s, and the teams observe whether the dashboards show it, whether the alerts (06-05) fire, whether the saga recovers and how long it takes them to understand it. What did not work is written down and turned into tasks. Tools like Chaos Mesh or LitmusChaos automate injection in Kubernetes; TechCorp starts by hand.

Summary table: pattern → problem → where it lives at TechCorp

Pattern	Problem it solves	Where it lives
Timeout with budget	Indefinite waits, resource exhaustion	`createHttpClient` (`AbortSignal.timeout`), `pg`/MongoDB pools, `HTTP_TIMEOUT_MS`
Retry with backoff + jitter	Transient errors	`retry()` in `@techcorp/common-http`; only idempotent/transient
Idempotency	"Did it run or not?"	`Idempotency-Key`, `processOnce`, `processed_events` (02-05, 03-01)
Circuit breaker	Cascading failure, dependency down	`createCircuitBreaker()` in `catalogClient`/`customersClient`; `circuit_breaker_state` metric
Bulkhead	A slow dependency exhausts the service	`p-limit` per dependency; separate pools; `resources.limits`
Fallback / degradation	Responding with something useful without the dependency	BFF without product names; Orders → 503 + `Retry-After`
Rate limiting / backpressure	Overload	Gateway 300/min and 429 (03-04); `prefetch(10)`
Fail-fast at startup	Inconsistent state	zod config (04-03); `/health/ready` 503 until dependencies are available
Retry queue with TTL + DLQ	Transient failures and poison messages in consumers	`<queue>.retry` (TTL 30 s), `<queue>.dlq`, `x-attempts`, `scripts/reprocessDlq.js`
Saga watchdog	Stuck sagas	`sagaWatchdog.js` in Orders (`PAYMENT_TIMEOUT`, 10 min)
Reconciliation	Whatever escapes all of the above	`CronJob reconcile-reservations` in Inventory
Graceful shutdown	Messages and requests lost during deployments	`SIGTERM` in `server.js`
Chaos engineering	Checking that all of the above works	Tests with toxiproxy, quarterly game days

Common Mistakes and Tips

Retrying everything. Retrying a POST without an idempotency key creates duplicates; retrying a 400 is wasted time; retrying against an overloaded service sinks it. Retry = transient and idempotent.
Retrying without jitter. All clients come back at once and create the "thundering herd".
A global breaker for all dependencies. A failure in Customers would also shut off Catalog. One per dependency.
Equal timeouts along the whole chain. The caller expires at the same time as the callee and nobody returns a useful response. Decreasing inward.
prefetch(1) "to process in order". A poison message blocks the queue. Ordering is handled differently (06-04).
DLQ with no owner and no procedure. Messages pile up for months; nobody knows whether they can be reprocessed. Every queue has an owning team (02-01) and its DLQ, alert and runbook (06-05).
Ignoring SIGTERM. Every deployment leaves half-done orders that the watchdog later cancels. Full graceful shutdown.
Confusing fallback with lying. Returning "stock available" because Inventory is not responding is not graceful degradation, it is an order that will be cancelled. Degrade only what does not compromise the business.
Tip: every pattern leaves a trail in logs (warn) and metrics (circuit_breaker_state, rabbitmq_queue_messages_ready{queue=~".*retry|.*dlq"}); if a pattern acts often, that is not a success of the pattern, it is a problem to fix in the dependency.
Tip: document per service, in its README, the table of dependencies with the timeout, retries, breaker and fallback of each one. It is the first page on-call opens.

Exercises

Exercise 1: Payments client toward the PSP

payments-service calls an external payment provider (POST /charges) that sometimes returns 503 and sometimes takes more than 5 s. Design the chain of patterns (timeout, retry, breaker, bulkhead) with concrete values and explain how you guarantee that a timeout does not produce a double charge.

Exercise 2: deciding the fate of three messages

On payments.stock (the stock.reserved consumer in Payments) three messages arrive that fail: (a) the PSP returned 503; (b) the message JSON has no orderId; (c) the PostgreSQL query fails with 40P01 deadlock_detected. State for each one whether it goes to .retry, to the .dlq or is retried within the process itself, and what gets logged.

Exercise 3: the watchdog and the reservation

An order has been in STOCK_RESERVED for 12 minutes because the payment.confirmed message is in orders.saga.dlq (Orders had a bug processing it). Describe the sequence of what the system does automatically, what on-call sees and what they must do, and what would have happened if the watchdog had acted at 16 minutes instead of 10.

Solutions

Exercise 1

Timeout: 5 s per attempt (the PSP is slow by nature; the gateway is not waiting on this call, it is a stock.reserved consumer).
Idempotency first: the PSP accepts an idempotency key; Idempotency-Key = orderId is sent. That way the second attempt after a timeout returns the same charge instead of a new one. In addition, before calling, Payments records charges(orderId, status='STARTED') in its DB; if the process dies and the message is redelivered, it first checks the status and asks the PSP by key before charging again.
Retry: retry(fn, { attempts: 3, baseMs: 500, maxMs: 5000 }) only for 503/429/timeout/ECONNRESET; a 402 (card declined) is permanent → payment.rejected.
Breaker: createCircuitBreaker({ name: 'psp', failureThreshold: 5, windowMs: 30000, openTimeMs: 60000 }); open → the message goes to payments.stock.retry (transient) without calling.
Bulkhead: p-limit(10) toward the PSP; the rest of Payments (status queries) does not depend on it.
Total budget for the message: 3 × 5 s + waits ≈ 20 s, well below the watchdog's 10 minutes.

Exercise 2

(a) 503 from the PSP: transient: true → publish to payments.stock.retry with x-attempts+1; warn "deferred retry" with eventId, orderId, attempt.
(b) no orderId: permanent validation error (poison message) → nack(requeue=false) → payments.stock.dlq on the first try; error "poison message, sending to DLQ" with the eventId and the reason. Someone will have to see why Inventory published that event (contract from 03-06).
(c) deadlock: transient and local → retry the transaction inside the process (retry with attempts: 3, baseMs: 50) without going through the queue; only if the attempts run out, to .retry. warn on each attempt with the SQL code.

Exercise 3

Sequence: (1) the orders.saga consumer fails to process payment.confirmed with a TypeError (permanent) → DLQ on the first try; error in Loki with eventId and orderId; rabbitmq_queue_messages_ready{queue="orders.saga.dlq"} = 1. (2) At 10 minutes the watchdog finds the order in STOCK_RESERVED, moves it to CANCELLED with PAYMENT_TIMEOUT and publishes order.cancelled. (3) Inventory releases the reservation (stock.released) and Payments, upon receiving order.cancelled for an order that did get charged, publishes payment.refunded (the compensation from 02-05); Notifications informs the customer. On-call sees the DLQ alert (06-05), reads the error in Loki, fixes the bug and deploys; then decides whether to reprocess the message: no in this case, because the order is already cancelled and refunded (reprocessing payment.confirmed on a CANCELLED order must be ignored by the state machine, and that is verified); it is discarded with --discard and the customer is contacted if appropriate. If the watchdog had acted at 16 minutes, the reservation would have expired on its own at 15 (expires_at) and the stock would have been released without a coordinated event; the order would have stayed in STOCK_RESERVED one more minute with a reservation that no longer existed and the refund would have arrived later: it works, but it is less clean. That is why the watchdog runs before the expiry.

Conclusion

This lesson has delivered on the promise of "designing for failure". Everything that goes over the network has a timeout with a decreasing budget (createHttpClient); transient errors on idempotent operations are retried with retry() (exponential backoff and jitter); dependencies that are down are isolated with a createCircuitBreaker() per dependency (circuit_breaker_state metric) and p-limit as a bulkhead; what can be degraded is degraded (BFF) and what cannot fails well (Orders → 503 with Retry-After); overload is held back with 429 and prefetch; startup fails fast and runtime tolerates. In the asynchronous world, every queue now has <queue>.retry with a 30 s TTL, <queue>.dlq after 5 attempts or on the first try for poison messages, and scripts/reprocessDlq.js; stuck sagas are cancelled by sagaWatchdog.js after 10 minutes with PAYMENT_TIMEOUT and the hourly reservation reconciliation is the last net; the process dies on unhandled promises and shuts down in order on SIGTERM; and all of it is tested with doubles, toxiproxy and game days. With resilience solved, the next question is one of capacity: when Black Friday arrives with the catalog at ×20 and orders at ×3, how many replicas are needed, how do they scale on their own and where are the bottlenecks? That is the topic of the next lesson: scalability and performance.

Error Handling and Recovery

Contents

Types of failure in a distributed system and the question "did it run or not?"

Timeouts: everything that goes over the network has a limit and a budget

Retries with exponential backoff and jitter: what to retry and what not to

Circuit breaker: cutting off before dragging the others down

Bulkhead: isolating dependencies

Fallback and graceful degradation; rate limiting and backpressure

Fail-fast at startup, tolerance at runtime

Recovery in the asynchronous world: retries, TTL queues and DLQ

Stuck sagas: the `PAYMENT_TIMEOUT` watchdog and reconciliation

Error handling in code: `BusinessError`, promises and `SIGTERM`

Resilience testing and basic chaos engineering

Summary table: pattern → problem → where it lives at TechCorp

Common Mistakes and Tips

Exercises

Exercise 1: Payments client toward the PSP

Exercise 2: deciding the fate of three messages

Exercise 3: the watchdog and the reservation

Solutions

Conclusion

Microservices Course

Module 1: Introduction to Microservices

Module 2: Microservice Design

Module 3: Communication between Microservices

Module 4: Implementing Microservices

Module 5: Deployment and Orchestration

Module 6: Monitoring and Maintenance

Module 7: Security in Microservices

Module 8: Case Studies and Practical Examples

Error Handling and Recovery

Contents

Types of failure in a distributed system and the question "did it run or not?"

Timeouts: everything that goes over the network has a limit and a budget

Retries with exponential backoff and jitter: what to retry and what not to

Circuit breaker: cutting off before dragging the others down

Bulkhead: isolating dependencies

Fallback and graceful degradation; rate limiting and backpressure

Fail-fast at startup, tolerance at runtime

Recovery in the asynchronous world: retries, TTL queues and DLQ

Stuck sagas: the PAYMENT_TIMEOUT watchdog and reconciliation

Error handling in code: BusinessError, promises and SIGTERM

Resilience testing and basic chaos engineering

Summary table: pattern → problem → where it lives at TechCorp

Common Mistakes and Tips

Exercises

Exercise 1: Payments client toward the PSP

Exercise 2: deciding the fate of three messages

Exercise 3: the watchdog and the reservation

Solutions

Conclusion

Microservices Course

Module 1: Introduction to Microservices

Module 2: Microservice Design

Module 3: Communication between Microservices

Module 4: Implementing Microservices

Module 5: Deployment and Orchestration

Module 6: Monitoring and Maintenance

Module 7: Security in Microservices

Module 8: Case Studies and Practical Examples

Stuck sagas: the `PAYMENT_TIMEOUT` watchdog and reconciliation

Error handling in code: `BusinessError`, promises and `SIGTERM`