In 04-02 we wrote catalog-service, in 04-04 orders-service, and between 03-04 and 07-01 the gateway. The other four services on the map from 01-05 —Inventory, Payments, Notifications and Customers— we have described many times (their tables in 02-04, their events in 02-05 and 03-02, their contracts in 03-01, their security in module 7) but never written. This lesson completes them. There is no new technique: every service uses the same node-service-template (04-01), the same modules from @techcorp/common-http and the same folder structure as Orders, so we will show in full, with comments, what is different in each one (its consumer and its key use case) and summarize in tables what is the same.
By the end we will have the whole system: the definitive event map, the saga sequence diagram with the six real services, the journey of Ana's order ord-88213 through every database and every queue with its timings, and the table of pacts and E2E tests each service contributes. Deploying and operating all of it is left for 08-03.
Contents
- Starting state: what is already written
- What the four new services have in common
inventory-servicepayments-servicenotifications-servicecustomers-service- Final event map and the saga with the six services
- The journey of
ord-88213through the complete system - Tests: which pacts and which E2E tests each service adds
- Starting state: what is already written
| Piece | Where it was written | What it contains | What it lacks for the complete system |
|---|---|---|---|
catalog-service (3001) |
04-02 (+ Redis in 06-04, HPA, authenticate({ optional: true }) in 07-01) |
createApp({ repository, logger, healthChecks }), productsService/productsRepository on MongoDB, GET /v1/products in three forms, scripts/seed.js (p-501, p-777, p-802) |
Nothing: it publishes product.updated (cache invalidation from 06-04, analytics); it was consumed only during the migration (08-01) |
orders-service (3002) |
04-04 (+ resilience 06-03, telemetry 06-02, authenticate/owner rule 07-01, audit 07-03) |
Order aggregate, createOrder with Idempotency-Key, saveWithEvents, outbox relay, sagaConsumer (orders.saga), customersConsumer (orders.customers), sagaWatchdog, GET /v1/orders/{id} with ETag |
For its four collaborators to actually exist |
| Gateway (8080) | 03-04, 07-01 | Routes `/api/v1/products | orders |
bff-mobile (3010) |
03-03, 03-04 | GraphQL over Catalog, Orders and Customers | Only referenced: we do not touch it |
@techcorp/common-http |
04-01, 03-02, 06-01, 06-03, 07-01, 07-03 | createLogger, requestIdMiddleware, sendProblem, errorMiddleware, createHealthRoutes, BusinessError, messaging/{topology,publisher,consumer,outbox,idempotency}, createHttpClient, retry, createCircuitBreaker, authenticate/requireRole/requireScope, createAudit, metrics |
Nothing |
About that last row, a reminder from 08-01 (phase 2): the outbox relay and processOnce were born in Orders (04-04) and moved to the library as soon as Inventory needed them. In this lesson we will use them from @techcorp/common-http/messaging/outbox (createOutbox(db) → enqueue(tx, events), createOutboxRelay) and .../messaging/idempotency (createIdempotency(db) → processOnce(eventId, consumer, fn)), with exactly the code from 04-04 §5 and §7.
- What the four new services have in common
All four are born from the template and share this shape; in each section we will only write the files marked with ★.
<name>-service/ ├── src/ │ ├── server.js app.js config.js health.js telemetry.js # 04-02, 04-03, 06-02: identical except for names │ ├── routes/*.js # only the endpoints in each service's table │ ├── use-cases/*.js ★ # the service's business logic │ ├── domain/*.js # pure states and rules │ ├── repositories/*.js # the service's SQL; no logic │ ├── clients/*.js (adapters to external systems) ★ # payment provider, email, Keycloak: ACL (02-03) │ ├── infra/postgres.js # the one from 04-04 §2, copied (30-line utility, 02-02 §6) │ └── messaging/<queue>Consumer.js ★ # on top of the library's createConsumer (06-03 §8) ├── migrations/NNN-*.sql scripts/migrate.js # 04-04 §2 ├── contracts/openapi.yaml contracts/asyncapi.yaml # 03-06 └── .github/workflows/ci.yml (9 lines: uses node-service-ci.yml@v1, 05-03 §11)
And the library's generic consumer, as it ended up in 06-03 §8, with the signature the three consumers in this lesson will use:
// @techcorp/common-http/messaging/consumer.js — signature (the body is the one from 06-03 §8)
// createConsumer({ channel, queue, routingKeys, handlers, idempotency, logger, prefetch = 10, maxAttempts = 5, retryTtlMs = 30000 })
// - declares <queue>, <queue>.retry (TTL) and <queue>.dlq with declareQueueWithRetry
// - for each message: parses the envelope {eventId, type, version, occurredAt, payload}; looks up handlers[type]
// - runs idempotency.processOnce(envelope.eventId, queue, (tx) => handler(envelope, tx)) → duplicates do not repeat effects
// - ack if it finishes; err.transient === true and x-attempts < maxAttempts → <queue>.retry; otherwise → <queue>.dlq
// - propagates traceparent (06-02) and requestId to the logs
// Returns { start, stop }.Error convention in the handlers: throw Object.assign(new Error('...'), { transient: true }) for "try again in 30 s" (DB or dependency down, event that arrives too early) and BusinessError or a plain error for "this will not fix itself" (straight to DLQ). It is the decision INC-2031 (06-05) turned into a rule.
inventory-service
inventory-servicePort 3006, Orders team, PostgreSQL inventory. It is the service that in the monolith was an UPDATE stock inside createOrder and is now the exclusive owner of the reserved <= quantity invariant (02-04 §7.2).
| Responsibility | Available stock, reservations with expiry, consumption and release; warehouse inbound stock |
| Endpoints | POST /v1/reservations (201 / 409 OUT_OF_STOCK; Idempotency-Key), DELETE /v1/reservations/{id} (204), GET /v1/stock/{productId} ({ productId, available, reserved }), PUT /v1/stock/{productId}/inbound (warehouse, operator role); all internal, with authenticate + requireRole('service' | 'operator') |
Consumes (queue inventory.orders) |
order.created → reserve; order.confirmed → consume; order.cancelled → release |
| Publishes | stock.reserved, stock.rejected, stock.released, stock.restocked (warehouse inbound; no consumer today) |
| Tables | stock, reservations, reservation_lines (02-04 §7.2), outbox, processed_events |
| Jobs | expireReservations every 60 s (ACTIVE reservations with expires_at < now()), CronJob reconcile-reservations (06-03 §9) |
config.js |
PORT=3006, INVENTORY_DB_URL (Secret inventory-db), RABBITMQ_URL (Secret inventory-rabbitmq), RESERVATION_TTL_S=900, EXPIRY_INTERVAL_MS=60000, OUTBOX_INTERVAL_MS, ORDERS_URL (reconciliation only), LOG_LEVEL, OTEL_*, KEYCLOAK_ISSUER/AUDIENCE |
The key use case is the reservation. Everything happens in one local transaction: locking the stock rows, checking, writing the reservation and the event in the outbox. It is T2 of the saga from 02-05.
// src/use-cases/reserveStock.js
const { randomUUID } = require('node:crypto');
function createReserveStockUseCase({ db, outbox, ttlSeconds, logger }) {
// Returns { status: 'RESERVED' | 'REJECTED', reservationId?, missing? }. Runs with the tx from processOnce
// (consumer) or with its own (POST /v1/reservations). Idempotent per orderId thanks to reservations.order_id UNIQUE.
return async function reserveStock({ orderId, lines, customerId, total }, tx) {
// 1. Is there already a reservation for this order? (redelivery, or repeated POST): return the same one, without touching stock
const previous = (await tx.query('SELECT reservation_id, status FROM reservations WHERE order_id = $1', [orderId])).rows[0];
if (previous) return { status: previous.status === 'ACTIVE' || previous.status === 'CONSUMED' ? 'RESERVED' : 'REJECTED', reservationId: previous.reservation_id };
// 2. Lock the stock rows involved, ALWAYS in the same order (by product_id) so that two orders
// with the same products do not lock each other crosswise (deadlock). FOR UPDATE: nobody else touches them until COMMIT.
const ids = [...new Set(lines.map((l) => l.productId))].sort();
const { rows: stock } = await tx.query('SELECT product_id, quantity, reserved FROM stock WHERE product_id = ANY($1) ORDER BY product_id FOR UPDATE', [ids]);
const byProduct = new Map(stock.map((s) => [s.product_id, s]));
// 3. Check ALL the lines before reserving any: the reservation is all or nothing
const missing = lines.filter((l) => { const s = byProduct.get(l.productId); return !s || s.quantity - s.reserved < l.quantity; }).map((l) => l.productId);
if (missing.length > 0) {
await outbox.enqueue(tx, [{ type: 'stock.rejected', payload: { orderId, reason: 'OUT_OF_STOCK', outOfStockProducts: missing } }]);
logger.info({ orderId, missing }, 'reservation rejected');
return { status: 'REJECTED', missing }; // no reservation: nothing to release later
}
// 4. Reserve: increment reserved (the CHECK reserved <= quantity from 02-04 is the last safety net) and create the reservation
const reservationId = `res-${randomUUID().slice(0, 8)}`;
const expiresAt = new Date(Date.now() + ttlSeconds * 1000);
for (const l of lines) await tx.query('UPDATE stock SET reserved = reserved + $1 WHERE product_id = $2', [l.quantity, l.productId]);
await tx.query(`INSERT INTO reservations (reservation_id, order_id, status, expires_at) VALUES ($1,$2,'ACTIVE',$3)`, [reservationId, orderId, expiresAt]);
for (const l of lines) await tx.query('INSERT INTO reservation_lines (reservation_id, product_id, quantity) VALUES ($1,$2,$3)', [reservationId, l.productId, l.quantity]);
// 5. The event goes out through the outbox in the SAME transaction (02-05 §7): either reservation + event, or nothing
await outbox.enqueue(tx, [{ type: 'stock.reserved', payload: { orderId, reservationId, lines, expiresAt: expiresAt.toISOString() } }]);
logger.info({ orderId, reservationId, expiresAt }, 'stock reserved');
return { status: 'RESERVED', reservationId };
};
}
module.exports = { createReserveStockUseCase };The other two use cases are short and symmetrical: consumeReservation(orderId, tx) moves the ACTIVE reservation to CONSUMED and runs UPDATE stock SET quantity = quantity - c, reserved = reserved - c per line (the definitive deduction that in the monolith sat inside the createOrder transaction); releaseReservation(orderId, tx) is literally the skeleton from 02-05 §5 (RELEASED, reserved - c, stock.released event). Both start with "if there is no reservation or it is not ACTIVE, do nothing": half of the idempotency. The consumer only routes:
// src/messaging/ordersConsumer.js
const { createConsumer } = require('@techcorp/common-http/messaging/consumer');
function createOrdersConsumer({ channel, idempotency, reserveStock, consumeReservation, releaseReservation, logger }) {
return createConsumer({
channel, queue: 'inventory.orders', idempotency, logger, prefetch: 10,
routingKeys: ['order.created', 'order.confirmed', 'order.cancelled'],
handlers: {
// The order.created payload (02-05 §4) carries lines[{productId, quantity}]: Inventory queries nobody
'order.created': (envelope, tx) => reserveStock({ orderId: envelope.payload.orderId, lines: envelope.payload.lines }, tx),
'order.confirmed': (envelope, tx) => consumeReservation(envelope.payload.orderId, tx),
'order.cancelled': (envelope, tx) => releaseReservation(envelope.payload.orderId, tx) // C2 of the saga; also for PAYMENT_TIMEOUT
}
});
}
module.exports = { createOrdersConsumer };And the expiry, the safety net independent of the Orders watchdog (02-05 §5, 06-03 §9): every minute, UPDATE reservations SET status='RELEASED' WHERE status='ACTIVE' AND expires_at < now() RETURNING … inside a transaction that also subtracts reserved and enqueues stock.released with reason: 'EXPIRED', with FOR UPDATE SKIP LOCKED so that two replicas do not step on each other. If a reservation for an order that is not CANCELLED ever expires, something is wrong in the saga: it is logged at error level and the reconciliation catches it. POST /v1/reservations reuses reserveStock inside db.transaction and answers 201 + Location or 409 OUT_OF_STOCK with outOfStockProducts in the problem (03-01 §5.5).
payments-service
payments-servicePort 3003, Payments & Communications team, PostgreSQL payments. It is the only service that talks to the external payment provider and the only one with Internet egress (07-04).
| Responsibility | Charge when stock is reserved, refund when a charged order is cancelled, receive confirmations from the payment provider, store tokenized payment methods |
| Endpoints | POST /v1/webhooks/payment-provider (HMAC, 07-02 §9, only referenced), POST /v1/payment-methods (customer: stores the tok_… the browser obtained from the payment provider, 07-03), GET /v1/payments?orderId= (operator), POST /v1/payments/{id}/refunds (operator; audited) |
Consumes (queue payments.stock) |
order.created → record the payment as PENDING (amount, customerId); stock.reserved → charge; order.cancelled → refund if it was CAPTURED, void if PENDING |
| Publishes | payment.confirmed, payment.rejected, payment.refunded |
| Tables | payments (payment_id, order_id UNIQUE, customer_id, amount, currency, status, provider_reference, attempts, created_at, updated_at), payment_methods (customer_id, provider_reference, is_default), received_webhooks, audit_log, outbox, processed_events |
config.js |
PORT=3003, PAYMENTS_DB_URL, RABBITMQ_URL, PAYMENT_PROVIDER_URL, PAYMENT_PROVIDER_API_KEY and PAYMENT_PROVIDER_WEBHOOK_SECRET (Secret payments-provider), PAYMENT_PROVIDER_TIMEOUT_MS=5000, PAYMENT_NEW_PROVIDER=false (flag), OUTBOX_INTERVAL_MS, LOG_LEVEL, OTEL_* |
A design nuance that 02-05 left implicit ("Payments charges without asking anyone") and here becomes explicit: Payments also listens to order.created, because stock.reserved is an Inventory event that should not carry amounts or customers. On receiving order.created it records the payment as PENDING with amount and customerId; on receiving stock.reserved it charges. If, due to concurrency, stock.reserved arrives before order.created has been processed, the handler throws a transient error and the message comes back in 30 s through the retry queue (06-03): ordering is solved with the mechanism that already exists, not with a new one. The payment statuses are the ones from 02-03 (AUTHORIZED, CAPTURED, REJECTED, REFUNDED) plus three implementation ones: PENDING (recorded, not yet talking to the payment provider), IN_PROGRESS (call under way) and VOIDED (cancelled before charging).
The adapter to the payment provider is an ACL with the four protections from 06-03 (ex. 1) and the branch by abstraction of the flag:
// src/clients/paymentProviderClient.js — ACL toward the payment provider: the rest of Payments only sees { ok, reference, reason }
const pLimit = require('p-limit');
const { createHttpClient, retry, createCircuitBreaker } = require('@techcorp/common-http');
function createPaymentProviderClient({ baseUrl, apiKey, timeoutMs, newProvider, logger, metrics }) {
const http = createHttpClient({ baseUrl, timeoutMs, name: 'payment-provider', headers: { Authorization: `Bearer ${apiKey}` } });
const breaker = createCircuitBreaker({ name: 'payment-provider', onChange: ({ state }) => metrics.breaker.set({ dependency: 'payment-provider' }, state) });
const limit = pLimit(10); // bulkhead: 10 simultaneous charges per replica
// Two providers, same interface: the PAYMENT_NEW_PROVIDER flag picks the translator (02-02 §4.2, 04-03 §8)
const translate = newProvider ? require('./translators/providerB') : require('./translators/providerA');
async function call(path, body, { orderId, requestId }) {
// Idempotency-Key = orderId: if the first attempt charged and the response was lost, the second returns the SAME charge
return limit(() => breaker.execute(() => retry(
() => http.request(path, { method: 'POST', body, requestId, headers: { 'Idempotency-Key': orderId } }),
{ attempts: 3, baseMs: 500, retryIf: (err) => err.transient === true } // 503/timeout yes; 402 (declined) no
)));
}
return {
// charge → { ok: true, reference } | { ok: false, reason: 'CARD_DECLINED' | 'INSUFFICIENT_FUNDS' | ... }
charge: async ({ orderId, amount, currency, customerReference, requestId }) =>
translate.charge(await call(translate.chargePath, translate.chargeBody({ orderId, amount, currency, customerReference }), { orderId, requestId })),
refund: async ({ orderId, reference, amount, requestId }) =>
translate.refund(await call(translate.refundPath(reference), { amount }, { orderId: `${orderId}-refund`, requestId })),
lookup: ({ orderId, requestId }) => call(translate.lookupPath, { key: orderId }, { orderId, requestId }).then(translate.charge)
};
}
module.exports = { createPaymentProviderClient };And the charging use case. Notice that the call to the payment provider goes outside the transaction (mistake 5 of createOrder in 01-05: never an external call inside a transaction) and how it survives a crash between the charge and the COMMIT:
// src/use-cases/chargeOrder.js
function createChargeOrderUseCase({ db, outbox, paymentProvider, logger }) {
// Invoked from the stock.reserved handler. It does NOT receive a tx: it manages its own short transactions,
// because in between there is an external call of up to 5 s that must not keep rows locked.
return async function chargeOrder({ orderId, requestId }) {
// 1. Short transaction: read the payment and mark it IN_PROGRESS (if already CAPTURED/REJECTED: duplicate, leave)
const payment = await db.transaction(async (tx) => {
const p = (await tx.query('SELECT * FROM payments WHERE order_id = $1 FOR UPDATE', [orderId])).rows[0];
if (!p) throw Object.assign(new Error('payment not yet recorded (order.created not processed)'), { transient: true }); // → retry in 30 s
if (p.status !== 'PENDING' && p.status !== 'IN_PROGRESS') return null; // already resolved: idempotent
await tx.query(`UPDATE payments SET status = 'IN_PROGRESS', attempts = attempts + 1, updated_at = now() WHERE payment_id = $1`, [p.payment_id]);
return { ...p, retryAfterCrash: p.status === 'IN_PROGRESS' };
});
if (!payment) return;
// 2. Outside the transaction: talk to the payment provider. If the process died with the payment IN_PROGRESS, first LOOK UP
// by the idempotency key (did it get charged?) before charging again (06-03 ex. 1).
const method = (await db.query('SELECT provider_reference FROM payment_methods WHERE customer_id = $1 AND is_default', [payment.customer_id])).rows[0];
let result;
if (!method) result = { ok: false, reason: 'NO_PAYMENT_METHOD' };
else if (payment.retryAfterCrash) result = await paymentProvider.lookup({ orderId, requestId }).catch(() => null);
if (!result) result = await paymentProvider.charge({ orderId, amount: payment.amount, currency: payment.currency, customerReference: method.provider_reference, requestId });
// (if the payment provider throws a transient error after exhausting retries, it propagates: the consumer sends it to .retry and the payment stays IN_PROGRESS)
// 3. Short transaction: result + event, together. UNIQUE(order_id) and the status guarantee "one order, one charge".
await db.transaction(async (tx) => {
if (result.ok) {
await tx.query(`UPDATE payments SET status = 'CAPTURED', provider_reference = $2, updated_at = now() WHERE payment_id = $1`, [payment.payment_id, result.reference]);
await outbox.enqueue(tx, [{ type: 'payment.confirmed', payload: { orderId, paymentId: payment.payment_id, amount: Number(payment.amount), currency: payment.currency } }]);
} else {
await tx.query(`UPDATE payments SET status = 'REJECTED', updated_at = now() WHERE payment_id = $1`, [payment.payment_id]);
await outbox.enqueue(tx, [{ type: 'payment.rejected', payload: { orderId, paymentId: payment.payment_id, reason: result.reason } }]);
}
});
logger.info({ orderId, paymentId: payment.payment_id, ok: result.ok, reason: result.reason }, 'charge resolved');
};
}
module.exports = { createChargeOrderUseCase };The payments.stock consumer has the same shape as Inventory's, with three handlers: 'order.created' → INSERT INTO payments (payment_id, order_id, customer_id, amount, currency, status) VALUES ('pay-…', $1, $2, $3, 'EUR', 'PENDING') ON CONFLICT (order_id) DO NOTHING; 'stock.reserved' → chargeOrder; 'order.cancelled' → refundIfApplicable: if the payment is CAPTURED, paymentProvider.refund and payment.refunded (C3 from 02-05, the rare branch: PAYMENT_TIMEOUT right after charging, or a customer cancellation within the window); if it is PENDING, VOIDED without calling anyone (OUT_OF_STOCK is the usual case). The webhook from 07-02 §9 does the same as step 3 of chargeOrder when the payment provider confirms asynchronously (deferred charges, chargebacks), and every manual refund goes through audit.record (07-03 §9).
notifications-service
notifications-servicePort 3005, Payments & Communications team. It was born from the monolith's services/email.js and is deliberately the simplest service: no business API, one consumer and one email adapter.
| Responsibility | Send the confirmation or cancellation email for each order, exactly once |
| Endpoints | Only /health/* and /metrics (not exposed on the gateway) |
Consumes (queue notifications.orders) |
order.created → store the recipient; order.confirmed → CONFIRMATION email; order.cancelled → CANCELLATION email with the reason |
| Publishes | Nothing (today). notification.sent is left as an extension |
| Tables | recipients (order_id PK, email, name, created_at) with 7-day retention (07-03 §8), deliveries (delivery_id, order_id, type, recipient, status, sent_at, UNIQUE (order_id, type)), processed_events |
config.js |
PORT=3005, NOTIFICATIONS_DB_URL, RABBITMQ_URL, EMAIL_PROVIDER=console|http, EMAIL_URL, EMAIL_API_KEY (Secret notifications-email), [email protected], EMAIL_TIMEOUT_MS=3000, LOG_LEVEL, OTEL_* |
About recipients: 07-03 decided that personal data travels only in order.created. That is why Notifications also subscribes to that event (the extra binding proposed by exercise 1 of 03-02) and stores email/name for seven days; order.confirmed and order.cancelled arrive without them. If an order.confirmed arrives before its order.created (possible with two replicas), the handler throws transient and waits 30 s: the same pattern as Payments.
// src/clients/emailClient.js — adapter with two modes: 'console' (development, E2E) and 'http' (fictional provider)
const { createHttpClient, retry } = require('@techcorp/common-http');
function createEmailClient({ provider, baseUrl, apiKey, sender, timeoutMs, logger }) {
if (provider === 'console') {
return { send: async (m) => { logger.info({ to: m.to, subject: m.subject }, '[console email]'); return { providerId: `console-${Date.now()}` }; } };
}
const http = createHttpClient({ baseUrl, timeoutMs, name: 'email', headers: { Authorization: `Bearer ${apiKey}` } });
return {
// send({ to, subject, html, text, reference }) → { providerId }. `reference` (orderId:type) is the provider's idempotency
// key: if we retry after a timeout, it does not send two emails. A 4xx (invalid address) is NOT transient → DLQ.
send: (m) => retry(() => http.request('/v1/messages', { method: 'POST', body: { from: sender, ...m }, headers: { 'Idempotency-Key': m.reference } }),
{ attempts: 2, baseMs: 300 })
};
}
module.exports = { createEmailClient };// src/use-cases/notifyOrder.js — the handler for order.confirmed / order.cancelled
const templates = require('../domain/templates'); // confirmation(o) and cancellation(o, reason) → { subject, html, text }; REASONS → readable text
const TYPE = { 'order.confirmed': 'CONFIRMATION', 'order.cancelled': 'CANCELLATION' };
function createNotifyOrderUseCase({ email, logger }) {
return async function notifyOrder(envelope, tx) {
const { orderId } = envelope.payload, type = TYPE[envelope.type];
// 1. Recipient stored when order.created was received. If it is not there yet: transient (arrived too early)
const recipient = (await tx.query('SELECT email, name FROM recipients WHERE order_id = $1', [orderId])).rows[0];
if (!recipient) throw Object.assign(new Error(`no recipient for ${orderId}`), { transient: true });
// 2. Business idempotency on top of the eventId one: one order, one email of each type (UNIQUE (order_id, type)).
// If an order.confirmed were re-emitted with ANOTHER eventId (reprocessed from the DLQ, 06-03), it would not duplicate either.
const { rowCount } = await tx.query(`INSERT INTO deliveries (delivery_id, order_id, type, recipient, status) VALUES (gen_random_uuid(), $1, $2, $3, 'SENDING') ON CONFLICT DO NOTHING`, [orderId, type, recipient.email]);
if (rowCount === 0) { logger.info({ orderId, type }, 'email already sent, ignored'); return; }
// 3. Send. If it fails transiently, the transaction ROLLS BACK (the SENDING row disappears) and the message
// comes back in 30 s; if the provider did send but we lost the response, its Idempotency-Key prevents the duplicate.
const message = envelope.type === 'order.confirmed' ? templates.confirmation({ ...envelope.payload, name: recipient.name }) : templates.cancellation({ ...envelope.payload, name: recipient.name }, envelope.payload.reason);
const { providerId } = await email.send({ to: recipient.email, reference: `${orderId}:${type}`, ...message });
await tx.query(`UPDATE deliveries SET status = 'SENT', provider_id = $3, sent_at = now() WHERE order_id = $1 AND type = $2`, [orderId, type, providerId]);
logger.info({ orderId, type, providerId }, 'email sent');
};
}
module.exports = { createNotifyOrderUseCase };Here the external call does go inside the processOnce transaction, unlike Payments. It is a conscious decision and it is worth understanding why it is acceptable: the transaction does not lock rows anyone else needs (only the order's own deliveries), the timeout is 3 s, and the benefit —that a send failure undoes the SENDING mark and the processed_events record at the same time— simplifies the code a lot. In Payments, with 5 s of payment provider and money involved, the same decision would be a bad one. The templates (domain/templates.js) translate the order.cancelled reasons into customer language: OUT_OF_STOCK → "the product has sold out", PAYMENT_REJECTED → "we were unable to charge you", PAYMENT_TIMEOUT → "the payment was not completed in time", CUSTOMER_CHANGED_MIND → "we have cancelled your order as you requested".
customers-service
customers-servicePort 3004, Shopping Experience team, PostgreSQL customers. It was the last one to be extracted (08-01, phase 6) and replaces the scripts/stubCustomers.js stub we had been dragging along since 04-04.
| Responsibility | Customer profile and addresses; sign-up linked to the Keycloak identity; right to erasure |
| Endpoints | POST /v1/customers (sign-up with the token of the user just registered in Keycloak), GET /v1/customers/{id} (the customer themselves, operator/admin or service with customers:read, 07-01 ex. 2), PUT /v1/customers/{id} (own or operator; publishes customer.updated), DELETE /v1/customers/{id} (own or admin; GDPR), `GET |
| Consumes | Nothing |
| Publishes | customer.updated (→ orders.customers, customers_ref replica), customer.deleted |
| Tables | customers (customer_id, keycloak_sub UNIQUE, email UNIQUE, name, active, created_at, updated_at), addresses (address_id, customer_id FK, street, postal_code, city, country, is_default), outbox, audit_log |
config.js |
PORT=3004, CUSTOMERS_DB_URL, RABBITMQ_URL, KEYCLOAK_URL=https://auth.techcorp.example, KEYCLOAK_REALM=techcorp, KEYCLOAK_ADMIN_CLIENT_ID=customers-service and KEYCLOAK_ADMIN_CLIENT_SECRET (Secret customers-keycloak; client credentials client with the realm's manage-users role), KEYCLOAK_ISSUER/AUDIENCE, OUTBOX_INTERVAL_MS, LOG_LEVEL, OTEL_* |
The use case that closes the loop with 07-01 is the sign-up: Ana registers in Keycloak (the realm's form), the store logs in and calls POST /v1/customers with her token; that token does not have the customerId claim yet. Customers creates c-…, writes the attribute in Keycloak and, on the next refresh, the token already travels with customerId: "c-1024".
// src/use-cases/registerCustomer.js
const { randomUUID } = require('node:crypto');
const { BusinessError } = require('@techcorp/common-http');
function createRegisterCustomerUseCase({ db, outbox, keycloak, logger }) {
// user = req.user from authenticate() (07-01): { sub, email, name }. The body may carry an initial address.
return async function registerCustomer({ name, address }, user) {
if (!user.sub) throw new BusinessError('UNAUTHENTICATED', 'sign-up without identity', 401);
// 1. Idempotent per keycloak_sub: repeating the POST returns the same customer (the browser often retries here)
const existing = (await db.query('SELECT customer_id FROM customers WHERE keycloak_sub = $1', [user.sub])).rows[0];
if (existing) return { customerId: existing.customer_id, repeated: true };
const customerId = `c-${randomUUID().slice(0, 8)}`;
// 2. Customer + address + event in ONE transaction; email from the token, not from the body (07-03: do not trust input)
await db.transaction(async (tx) => {
await tx.query(`INSERT INTO customers (customer_id, keycloak_sub, email, name, active) VALUES ($1,$2,$3,$4,true)`, [customerId, user.sub, user.email, name]);
if (address) await tx.query(`INSERT INTO addresses (address_id, customer_id, street, postal_code, city, country, is_default) VALUES ($1,$2,$3,$4,$5,$6,true)`,
[`addr-${randomUUID().slice(0, 6)}`, customerId, address.street, address.postalCode, address.city, address.country ?? 'ES']);
await outbox.enqueue(tx, [{ type: 'customer.updated', payload: { customerId, name, email: user.email, updatedAt: new Date().toISOString() } }]);
});
// 3. Outside the transaction: write the attribute in Keycloak (external call; retryable and idempotent: PUT of the attribute).
// If it fails, the customer exists anyway and a 'syncKeycloak' job retries it: the token without customerId only
// prevents creating orders for a few minutes (403 in Orders, 07-01), it does not buy anything wrong.
try { await keycloak.setAttribute(user.sub, 'customerId', customerId); }
catch (err) { logger.error({ err, customerId, sub: user.sub }, 'could not write customerId to Keycloak; pending sync'); await db.query('INSERT INTO keycloak_pending (customer_id) VALUES ($1) ON CONFLICT DO NOTHING', [customerId]); }
logger.info({ customerId }, 'customer registered');
return { customerId, repeated: false };
};
}
module.exports = { createRegisterCustomerUseCase };The other two use cases, summarized: PUT /v1/customers/{id} validates with zod (07-03 §3: name ≤ 120 characters, no HTML), applies the owner-or-operator rule, runs UPDATE customers SET name, updated_at = now() and enqueues customer.updated in the same transaction; the orders.customers consumer from 04-04 applies it to customers_ref with the updatedAt protection from exercise 2 of 04-04. DELETE /v1/customers/{id} (GDPR, 07-03 §8) anonymizes instead of deleting (email = 'anon-<hash>@deleted', name = 'Deleted customer', active = false, addresses deleted), enqueues customer.deleted, disables the user in Keycloak, and records CUSTOMER_DELETED in audit_log with the sub of whoever requested it; Orders consumes customer.deleted on orders.customers to delete the customers_ref row and anonymize addresses on old orders. keycloakClient (src/clients/keycloakClient.js) is createHttpClient + createTokenProvider (07-01 §8) over the admin API (PUT /admin/realms/techcorp/users/{sub} with attributes.customerId).
- Final event map and the saga with the six services
With the four services written, this is the definitive map. Compared with the table in 03-02 there are two more bindings (order.created on payments.stock and on notifications.orders), one more consumer of customer.* (analytics, 08-01 §8) and every compensation event with a real producer:
| Event | Producer | Consumers (queue) | Essential payload |
|---|---|---|---|
order.created |
Orders | Inventory (inventory.orders), Payments (payments.stock), Notifications (notifications.orders), analytics |
orderId, customerId, customer{email,name}, shippingAddress, lines[], total |
stock.reserved |
Inventory | Orders (orders.saga), Payments (payments.stock) |
orderId, reservationId, lines, expiresAt |
stock.rejected |
Inventory | Orders (orders.saga) |
orderId, reason: OUT_OF_STOCK, outOfStockProducts[] |
payment.confirmed |
Payments | Orders (orders.saga) |
orderId, paymentId, amount, currency |
payment.rejected |
Payments | Orders (orders.saga) |
orderId, paymentId, reason |
order.confirmed |
Orders | Inventory, Notifications, analytics | orderId, customerId, lines, total (no personal data) |
order.cancelled |
Orders (saga consumer, watchdog, DELETE /v1/orders/{id}) |
Inventory, Payments, Notifications, analytics | orderId, reason |
stock.released |
Inventory | (nobody today; analytics) | orderId, reservationId, reason? |
payment.refunded |
Payments | (nobody today; analytics) | orderId, paymentId, amount |
customer.updated / customer.deleted |
Customers | Orders (orders.customers), analytics |
customerId, name, email, updatedAt / customerId |
product.updated |
Catalog | analytics | productId, name, price, category, published |
sequenceDiagram
autonumber
actor Ana
participant GW as gateway :8080
participant ORD as orders-service :3002
participant CUS as customers-service :3004
participant CAT as catalog-service :3001
participant MQ as RabbitMQ techcorp.events
participant INV as inventory-service :3006
participant PAY as payments-service :3003
participant PSP as Payment provider
participant NOT as notifications-service :3005
Ana->>GW: POST /api/v1/orders (JWT customerId=c-1024, Idempotency-Key)
GW->>ORD: POST /v1/orders + X-User-*
par synchronous validations
ORD->>CUS: GET /v1/customers/c-1024 (service token)
ORD->>CAT: GET /v1/products?ids=p-501,p-777
end
ORD->>ORD: tx: order PENDING + outbox(order.created)
ORD-->>Ana: 202 Location /v1/orders/ord-88213
ORD--)MQ: order.created (relay)
MQ--)INV: inventory.orders
MQ--)PAY: payments.stock → payment PENDING
MQ--)NOT: notifications.orders → recipient
INV->>INV: tx: FOR UPDATE, reservation ACTIVE + outbox(stock.reserved)
INV--)MQ: stock.reserved
MQ--)ORD: orders.saga → STOCK_RESERVED
MQ--)PAY: payments.stock → charge
PAY->>PSP: POST charge (Idempotency-Key = orderId)
PSP-->>PAY: ok, reference
PAY->>PAY: tx: CAPTURED + outbox(payment.confirmed)
PAY--)MQ: payment.confirmed
MQ--)ORD: orders.saga → PAID → CONFIRMED + outbox(order.confirmed)
ORD--)MQ: order.confirmed
MQ--)INV: reservation CONSUMED, quantity -= c
MQ--)NOT: CONFIRMATION email
NOT->>Ana: "Order ord-88213 confirmed"
- The journey of
ord-88213 through the complete system
ord-88213 through the complete systemAna's order (c-1024, p-501 ×1 and p-777 ×2, €79.70), with fictional timings consistent with the trace from 06-02 §8 (gateway 3 ms, Orders 180 ms, Catalog 40 ms, Customers 25 ms):
| Instant (UTC) | Service | What happens | Row / message that appears |
|---|---|---|---|
| 10:42:00.000 | gateway | POST /api/v1/orders; valid JWT, customerId=c-1024 = body |
log requestId=7f3c…, traceparent |
| 10:42:00.003 | orders | Promise.all: Customers (25 ms) and Catalog (40 ms) |
— |
| 10:42:00.180 | orders (PG orders) |
COMMIT |
orders(ord-88213, PENDING, 79.70), 2 order_lines, customers_ref(c-1024), outbox(evt-a1, order.created), idempotency_keys |
| 10:42:00.183 | gateway → Ana | 202 Accepted, Location: /v1/orders/ord-88213 |
— |
| 10:42:00.420 | orders (relay) | publishes and marks | outbox.published_at; on techcorp.events → 4 queues |
| 10:42:00.470 | payments (PG payments) |
order.created |
payments(pay-9001, ord-88213, c-1024, 79.70, PENDING), processed_events(evt-a1, payments.stock) |
| 10:42:00.475 | notifications | order.created |
recipients(ord-88213, [email protected], Ana Ruiz) |
| 10:42:00.490 | inventory (PG inventory) |
FOR UPDATE p-501, p-777; reservation |
stock.reserved p-501 +1, p-777 +2; reservations(res-4471, ACTIVE, expires 10:57:00), 2 reservation_lines, outbox(evt-b2, stock.reserved), processed_events(evt-a1, inventory.orders) |
| 10:42:00.720 | inventory (relay) | publishes | stock.reserved → orders.saga, payments.stock |
| 10:42:00.760 | orders | orders.saga |
orders.status = STOCK_RESERVED, processed_events(evt-b2, orders.saga); Ana's GET would return this status |
| 10:42:00.790 | payments | payments.stock: IN_PROGRESS; call to the payment provider (1.2 s) |
payments.status = IN_PROGRESS, attempts = 1 |
| 10:42:02.010 | payments | OK response ch_7f3a… |
payments.status = CAPTURED, provider_reference, outbox(evt-c3, payment.confirmed), processed_events(evt-b2, payments.stock) |
| 10:42:02.260 | payments (relay) → orders | payment.confirmed on orders.saga |
orders.status = CONFIRMED (PAID → CONFIRMED in the same transaction, 04-04 §8), outbox(evt-d4, order.confirmed), processed_events(evt-c3, orders.saga) |
| 10:42:02.500 | orders (relay) | publishes | order.confirmed → inventory.orders, notifications.orders, analytics |
| 10:42:02.540 | inventory | consume | reservations(res-4471) = CONSUMED; stock p-501 quantity −1, reserved −1; p-777 −2/−2 |
| 10:42:02.900 | notifications | deliveries(ord-88213, CONFIRMATION, SENT, provider_id); Ana receives the email |
|
| 10:42:02.9 | metrics | saga_duration_seconds observes 2.7 s; orders_created_total +1 |
"Orders saga" dashboard (06-01) |
Compare it with the diagram in 01-05 §5: they are the same eight business operations, but in five local transactions across four databases, joined by six messages, each with its idempotency mark. And if at 10:42:00.490 there had been no stock: stock.rejected → CANCELLED (OUT_OF_STOCK) at 10:42:00.8, order.cancelled → Payments voids the PENDING payment without calling anyone, Notifications sends "product sold out", Inventory has no reservation to release. And if the payment provider declined: payment.rejected → CANCELLED (PAYMENT_REJECTED) → Inventory releases (stock.released), Notifications "we were unable to charge you".
- Tests: which pacts and which E2E tests each service adds
Each service comes with the pyramid from 04-05 (domain unit tests, component tests with createApp and doubles, integration with Testcontainers). What each one adds to the whole are the contracts and the E2E tests:
| Service | Pacts (consumer → provider) | E2E it adds in platform/tests/e2e/ |
|---|---|---|
| Inventory | HTTP: reconcile-reservations → Orders GET /v1/orders/{id} (state 'ord-… exists and is CANCELLED'). Messages: Inventory as consumer of order.created (Orders verifies that its payload satisfies lines[{productId, quantity}]); Inventory as provider of stock.reserved for Orders and Payments (the corrective action from INC-2031: lines never null) |
outOfStock.e2e.test.js: order with p-802 (stock 0) → CANCELLED with reason OUT_OF_STOCK in < 5 s and GET /v1/stock/p-802 with nothing reserved |
| Payments | Messages: consumer of order.created and stock.reserved; provider of payment.confirmed/payment.rejected for Orders. Toward the payment provider: no Pact (third party) → component tests against a fake server with the 200/402/503/timeout/duplicate cases (06-03 §11) |
paymentRejected.e2e.test.js: fictional payment method tok_decline → CANCELLED (PAYMENT_REJECTED), stock.released and CANCELLATION email in console mode |
| Notifications | Messages: consumer of order.created, order.confirmed, order.cancelled (Orders verifies) |
Covered by the previous ones by checking deliveries (one row per order and type) |
| Customers | HTTP: Orders → Customers GET /v1/customers/{id} (the pact from exercise 2 of 04-05, now verified by the real service instead of the stub); Customers as provider of customer.updated/customer.deleted for Orders |
customerDeleted.e2e.test.js: DELETE /v1/customers/{id} → customers_ref disappears in Orders and GET /v1/customers/{id} → 404 |
| (already existing) | Orders → Catalog GET /v1/products?ids= (04-05) |
createOrder.e2e.test.js (04-05, 05-01): now reaches CONFIRMED with the real services, without publishEvent.js |
The message pacts use the same tool (@pact-foundation/pact, MessageConsumerPact / MessageProviderPact) and the same Broker with can-i-deploy from 05-03 §4: if Orders changes the order.created payload, Orders' CI knows it breaks three consumers before deploying. It is the definitive answer to cause 1 of INC-2031.
Common Mistakes and Tips
- Copying the consumer from 04-04 into every service instead of using
createConsumer. Four copies of thex-attempts, retry and DLQ handling diverge within a month. It is technical code: library (02-02 §6). - Locking stock rows in a different order on two paths (
ORDER BY product_idin the reservation and no order in the warehouse inbound). Two transactions wait on each other crosswise and PostgreSQL kills one with40P01. Same order, always. - The payment provider call inside the transaction "to keep it simple". Five seconds with the
paymentsrow locked and, worse, aROLLBACKthat undoes the record of a charge the payment provider did make. Short transactions around it; look up by idempotency key when in doubt. - Relying only on
eventIdto avoid sending two emails. A reprocess from the DLQ or a re-emittedorder.confirmedcarries anothereventId. Business idempotency (UNIQUE (order_id, type)) is what protects the customer. - Putting
emailinorder.confirmed"because Notifications needs it". 07-03 restricted it toorder.created; the solution is for Notifications to keep it for seven days, not to reopen the decision. - Writing the attribute in Keycloak inside the sign-up transaction. It is an external call; if Keycloak is slow, the
INSERTwaits; if it fails, the customer does not exist and the user cannot retry either (they are already registered in Keycloak). Outside, retryable, with a pending queue. - Tip: when you write the fourth service with the same template, measure how long it takes you. If it is hours and not days, phase 0 from 08-01 and Luis's rule have worked; if it is days, something in the template or the library is missing and it must be pushed up before the fifth.
Exercises
Exercise 1: The customer cancels in time
Add customer cancellation to orders-service (DELETE /v1/orders/{id} from 03-01, reason CUSTOMER_CHANGED_MIND), allowed only in CONFIRMED for 30 minutes. Indicate which transition you would add to the state machine from 02-05, which event is published and what each of the four services in this lesson does on receiving it (one line per service, with the table or call involved).
Exercise 2: stock.reserved before order.created
In Payments, stock.reserved can arrive before order.created has been processed. Explain (a) why it is possible even though Orders publishes order.created before stock.reserved exists; (b) what happens step by step with this lesson's design (transient, retry queue, x-attempts); (c) what alternative there would be if, instead of a transient error, we wanted to resolve it without waiting 30 s, and what it costs.
Exercise 3: A message pact
Write, in pseudocode or with the Pact API for messages, the contract Payments declares as consumer of stock.reserved: which fields it requires, with which matchers, and which provider state Inventory should prepare to verify it. Explain what this pact would have detected on July 22, 2026 (INC-2031).
Solutions
Exercise 1. Transition CONFIRMED --customer.cancel--> CANCELLED (with the now() - updated_at < 30 min check in the use case, not in the transitions table), reason CUSTOMER_CHANGED_MIND, ORDER_CANCELLED audit entry (07-03) and order.cancelled { orderId, reason } event through the outbox. Reactions: Inventory (inventory.orders): the reservation is already CONSUMED, so releaseReservation does nothing; a new case is needed, restockOnCancellation: if the reservation is CONSUMED, UPDATE stock SET quantity = quantity + c per line and stock.restocked (or mark it RETURNED); Payments (payments.stock): refundIfApplicable finds the payment CAPTURED → paymentProvider.refund with key ord-…-refund → REFUNDED + payment.refunded (branch C3 from 02-05, now a common one); Notifications: CANCELLATION email with the CUSTOMER_CHANGED_MIND text (already in the templates), idempotent by (order_id, CANCELLATION); Customers: nothing (it does not consume order events). And analytics records the cancellation with its reason.
Exercise 2. (a) order.created enters payments.stock before stock.reserved, but the queue has several messages in flight (prefetch 10) and two replicas: replica A takes order.created and its transaction takes 40 ms; replica B takes stock.reserved 30 ms later and runs it before A COMMITs. Queue order is delivery order, not completion order. (b) chargeOrder does not find the row → throws { transient: true } → createConsumer publishes the message to payments.stock.retry with x-attempts: 1 and acks the original → 30 s later it comes back through techcorp.events.retry → now the row exists → normal charge. Cost: 30 more seconds of saga for that order (within the 60 s SLO from 06-05, but counting); with maxAttempts 5 there would be 2 minutes of margin before the DLQ. (c) Alternative: in the stock.reserved handler, if there is no payment, create the PENDING row from the event's own data —which would require stock.reserved to carry amount and customerId, coupling Inventory to data that is not its own— or query GET /v1/orders/{id} synchronously (temporal coupling, another HTTP client, another pact). Both avoid the wait at the cost of more coupling; TechCorp accepts the occasional 30 s because they are rare (only under real concurrency) and the mechanism already exists.
Exercise 3.
// payments/tests/contract/stockReserved.consumer.pact.test.js (outline)
const messagePact = new MessageConsumerPact({ consumer: 'payments-service', provider: 'inventory-service', dir: 'pacts' });
await messagePact
.given('an ACTIVE reservation exists for ord-88213') // provider state Inventory prepares with its in-memory repository
.expectsToReceive('stock.reserved for an order with lines')
.withContent({ eventId: like('evt-b2'), type: 'stock.reserved', version: integer(1), occurredAt: iso8601DateTime(),
payload: { orderId: regex(/^ord-[0-9a-f]{8}$/, 'ord-88213'), reservationId: like('res-4471'),
lines: eachLike({ productId: like('p-501'), quantity: integer(1) }, { min: 1 }), expiresAt: iso8601DateTime() } })
.verify(async (message) => stockReservedHandler(message.contents, fakeTx)); // Payments' REAL handler with the pact's payloadIt requires orderId in the format from 07-03, reservationId, lines as an array of at least one element with integer productId and quantity, and expiresAt as a date; with like/eachLike instead of exact values (04-05: types and shape, not data). Inventory, as provider, verifies in its CI that its publisher produces a message that satisfies that contract for that state. On July 22, Inventory 2.3.0 published lines: null: the provider verification would have failed in Inventory's CI before building the image, can-i-deploy would have said no, and payments.stock would not have jammed. (Even though the Payments handler that failed with TypeError was the stock.reserved one, which today does not use lines; the pact still documents what Payments tolerates and what it does not.)
Conclusion
TechCorp's system is complete. To the three components already written —catalog-service (04-02), orders-service (04-04) and the gateway (03-04/07-01)— we have added, with the same template and the same library, the four that were missing: Inventory, owner of the reserved <= quantity invariant, with the all-or-nothing reservation under FOR UPDATE in a fixed order, stock.reserved/stock.rejected through the outbox, idempotent consumption and release, and reservation expiry as an independent safety net; Payments, with the paymentProviderClient ACL (idempotency key = orderId, retries only for transient errors, circuit breaker, bulkhead, two providers behind a flag), short transactions around the external call, UNIQUE (order_id) and the refund branch; Notifications, with seven-day recipients to honor the GDPR decision from 07-03, deliveries with UNIQUE (order_id, type) as business idempotency and the email adapter with console mode; and Customers, with the sign-up that writes the customerId claim in Keycloak outside the transaction, customer.updated for the Orders replica and customer.deleted for the right to erasure. The event map is now closed (eleven types, five queues plus the analytics one), the saga has been drawn with the six real services, ord-88213 has crossed four databases and six messages in 2.9 seconds, and each service has contributed its pacts (HTTP and message) and its E2E tests.
All of this is repositories with code and green tests. The next lesson takes them to a cluster: the complete techcorp/platform repository, the compose.yaml with the six services and all the infrastructure, the startup order on a fresh cluster, the smoke test with a Keycloak token, and day-to-day operations —an end-to-end Orders deployment, the Black Friday campaign, an incident in the DLQ, a secret rotation, the Node upgrade and a contract evolution— with their approximate monthly cost.
Microservices Course
Module 1: Introduction to Microservices
- Basic Concepts of Microservices
- Advantages and Disadvantages of Microservices
- Comparison with the Monolithic Architecture
- When to Adopt Microservices: Decision Criteria
- The Course Case Study: TechCorp's Online Store
Module 2: Microservice Design
- Microservice Design Principles
- Decomposing Monolithic Applications
- Defining Bounded Contexts
- Data Management: One Database per Service
- Distributed Consistency: Sagas, CQRS and Event Sourcing
Module 3: Communication between Microservices
- RESTful APIs
- Asynchronous Messaging
- Communication Protocols: gRPC, GraphQL
- API Gateway and Backend for Frontend
- Service Discovery and Load Balancing
- API Contracts and Versioning
Module 4: Implementing Microservices
- Choosing Technologies and Tools
- Building a Simple Microservice
- Configuration Management
- Hands-On Integration: Consuming APIs and Publishing Events
- Testing Microservices: Unit, Integration and Contract Tests
Module 5: Deployment and Orchestration
- Containers and Docker
- Orchestration with Kubernetes
- CI/CD for Microservices
- Deployment Strategies: Rolling, Blue-Green and Canary
- Service Mesh: Istio and Linkerd
Module 6: Monitoring and Maintenance
- Monitoring and Logging
- Distributed Tracing with OpenTelemetry
- Error Handling and Recovery
- Scalability and Performance
- SLOs, Alerts and Incident Management
Module 7: Security in Microservices
- Authentication and Authorization
- Communication Security
- Security Practices
- Container and Kubernetes Security