In the previous lesson we fixed TechCorp's REST contracts, but we made it clear that the synchronous calls would be few: Orders → Catalog and Orders → Customers. The rest of the "a customer places an order" flow (reserving stock, charging, confirming, notifying) was designed in 02-05 as a choreography saga in which the services do not call each other, but rather publish and consume events. Until now those events (order.created, stock.reserved, payment.confirmed, order.confirmed, order.cancelled) have been names in a diagram. This lesson turns them into real messages that travel through RabbitMQ.

We will first look at when asynchronous communication is appropriate and which coupling it removes; then the essential vocabulary (message, event and command, queue and pub-sub, broker, ack, redelivery, dead-letter queue, ordering and delivery guarantees); next RabbitMQ in detail (exchanges, routing keys, queues, bindings) with TechCorp's concrete topology; the amqplib code to publish the standard event envelope and to consume with prefetch, ack/nack and forwarding to the DLQ; how it hooks into the outbox and the idempotency from 02-05; and a comparison of RabbitMQ with Kafka and managed cloud queues that justifies TechCorp's choice. The end-to-end integration of a real service (with its outbox, its consumers starting alongside Express) belongs to 04-04, and deploying the broker to module 5.

1. Synchronous versus asynchronous: which coupling we remove
2. Messaging vocabulary
3. Delivery guarantees: at-most-once, at-least-once, exactly-once
4. RabbitMQ: exchanges, queues, routing keys and bindings
5. TechCorp's topology
6. Publishing an event with amqplib
7. Consuming events: prefetch, ack, nack and DLQ
8. Hooking into outbox and idempotency
9. RabbitMQ versus Kafka and managed queues

When Orders calls GET /products?ids= and waits for the response, Orders and Catalog have to be alive at the same instant: this is the temporal coupling from 02-01. With six services chained in the order flow, that coupling would be fatal: the availability of the whole would be the product of the six, and an outage in Notifications would prevent orders from being created. Asynchronous messaging breaks that chain: Orders leaves the event in the broker and moves on; Inventory will pick it up when it can, even if that is a minute later.

The practical rule TechCorp will apply: asynchronous between services by default; synchronous only when the response is essential to answer the user. It is the same conclusion as 02-02, now with the technical justification.

• Message. The unit of data that travels through the broker: some headers and a body (at TechCorp, the JSON envelope from 02-05).
• Event versus command. An event describes something that has already happened (order.created); whoever publishes it expects nothing from anyone, and there may be zero or many consumers. A command is an instruction aimed at a specific receiver (reserve-stock): whoever sends it expects someone to execute it. The choreography saga from 02-05 uses only events; an orchestration saga would use commands. The distinction matters because it changes who decides: with events, the consumer decides whether it is interested; with commands, the sender decides whom to address.
• Producer (publisher) and consumer (subscriber). Who sends and who receives. A service is usually both: Inventory consumes order.created and produces stock.reserved.
• Broker. The intermediary that receives, stores and delivers messages: RabbitMQ, Kafka, SQS. It is the "dumb pipe" from 02-01: it routes, it does not decide.
• Queue. A FIFO buffer from which one or more consumers compete for messages: each message is processed by one instance. Ideal for spreading work across replicas of the same service.
• Topic / pub-sub. A message is copied to every interested subscriber. Ideal for events: order.confirmed is of interest to Notifications and to Inventory, and both must receive it. In RabbitMQ, the combination "one exchange that fans out to several queues, each queue with the replicas of one service competing" gives both behaviors at once.
• Ack / nack. The consumer confirms (acknowledges) that it has processed the message; until then the broker considers it pending. If the consumer rejects it (negative ack) or dies without confirming, the broker redelivers it, to the same or to another instance.
• Dead-letter queue (DLQ). The queue a message goes to when it could not be processed (rejected without requeue, expired or evicted by a limit). It avoids the "poison message" that is redelivered forever and lets you inspect it and reprocess it by hand.
• Ordering. RabbitMQ preserves order within a queue for a single consumer; with several replicas consuming in parallel, global order is not guaranteed. That is why the design from 02-05 makes every event self-contained and has the order state machine reject out-of-order transitions.

The realistic conclusion, already anticipated by 02-05: we choose at-least-once + idempotency. The broker guarantees that no event is lost (even if some arrive twice), and the consumer guarantees that processing the same eventId twice has no effect (processOnce() with the processed_events table). It is the only combination that works with one database per service.

RabbitMQ implements the AMQP 0-9-1 protocol, whose model has four pieces:

• Exchange. The producer never publishes directly to a queue: it publishes to an exchange with a routing key (a string like order.created). The exchange decides which queues to copy the message to depending on its type:
  • direct: to the queues whose binding matches the routing key exactly.
  • fanout: to every bound queue, ignoring the routing key.
  • topic: to the queues whose binding pattern matches the routing key, with wildcards: * stands for exactly one word (separated by dots) and # for zero or more. order.* matches order.created and order.cancelled; # matches everything.
  • headers: routes by headers; rarely used.
• Queue. Where messages wait. It can be durable (survives a broker restart) and hold persistent messages (written to disk). For at-least-once you need both.
• Binding. The rule that ties an exchange to a queue: "messages whose routing key matches order.created go to the inventory.orders queue".
• Channel. A logical connection multiplexed inside a TCP connection; each thread or consumer uses its own.

A message arrives at an exchange with its routing key, the exchange copies it to every queue whose binding matches (that is how pub-sub is achieved), and within each queue the replicas of the consuming service compete (that is how load distribution is achieved). One and the same message can end up in three queues and be processed once in each.

Decisions:

• A single exchange of type topic, named techcorp.events, durable. Every service publishes to it.
• The routing key is equal to the event type: order.created, stock.reserved, payment.confirmed, etc. That way the event name and its routing are the same thing and there are not two vocabularies to maintain.
• One queue per consuming service, named <consumer>.<topic>, with bindings to the events it cares about. The replicas of a service share its queue (load distribution); different services have different queues (each receives its own copy).
• Each queue has an associated DLQ (<queue>.dlq) through a techcorp.events.dlx exchange of type direct.

Notice that Orders publishes order.confirmed upon receiving payment.confirmed, and that event is consumed by two different queues (Notifications and Inventory): the copy is made by the exchange, not by Orders. Orders neither knows nor cares how many consumers there are: that is the "conformist" role of Notifications and the asymmetry of the context map from 02-03 turned into topology.

About orders.saga with the stock.* and payment.* bindings: it is convenient, but it would also receive stock.released and payment.refunded, which Orders does not need. In practice the four explicit bindings from the table are declared; the wildcard appears in the diagram only for brevity.

amqplib is the standard Node.js library for AMQP 0-9-1. First, the topology declaration. Each service declares at startup what it uses (the exchange and its own queues); assert* is idempotent: if it already exists with the same parameters, it does nothing.

Now the publishing. The function receives the standard event envelope from 02-05 already built (eventId, type, version, occurredAt, payload) and sends it:

And its use from the Orders outbox relay, with the order.created event for order ord-88213 (the payload JSON is the one we fixed in 02-05):

Three details that make the difference between "works on my machine" and "does not lose orders":

1. persistent: true and durable: true queues go together. A persistent message in a non-durable queue is lost on restart all the same; a durable queue with non-persistent messages, too.
2. Publisher confirms. With createChannel(), publish is "fire and forget": if RabbitMQ goes down right then, the message is lost without an error. With createConfirmChannel() the broker confirms each publication and we can wait with await channel.waitForConfirms() before marking the outbox row as sent. This is what the relay will use in 04-04.
3. The eventId travels in the body and in messageId. In the body because it is part of the event contract; in the header because the RabbitMQ console and the DLQ tools show it without opening the JSON.

The Inventory consumer for the inventory.orders queue:

And the handlers that Inventory registers (signature only; the reservation logic belongs to module 4):

Points worth understanding well:

• prefetch(10) limits the in-flight work per replica. A low value spreads work better across replicas and prevents a dying process from dragging hundreds of messages into redelivery; a high value increases throughput. Ten is a reasonable starting point for handlers that touch the database.
• ack after processing is what gives at-least-once. If the process dies between handler() and ack, RabbitMQ redelivers and processOnce prevents the double effect. Never ack at the beginning "so it doesn't get stuck": that is at-most-once with a pretty name.
• nack(msg, false, requeue): the second argument (allUpTo) also rejects all previous unacked messages; almost always false. The third decides between requeuing (true) and discarding/dead-lettering (false).
• Requeuing without a limit is an infinite loop. A message that always fails would go back to the head of the queue and block the rest. That is why it is requeued at most once (msg.fields.redelivered says whether it already was) and then goes to the DLQ. Retries with growing backoff are covered in 06-03; RabbitMQ does not provide them out of the box.
• The DLQ is not consumed automatically. Someone (an alert from 06-05 and a person) looks at inventory.orders.dlq, understands why it failed and decides whether to reprocess (move the message back) or discard.

With what we have seen, the complete chain of guarantees for "a customer places an order" looks like this, and it is worth seeing it all together even though each piece was designed in 02-05:

None of this is exotic: it is at-least-once at every hop plus deduplication by eventId at the destination. It is the price of not having distributed transactions, and it is cheap.

Why TechCorp chooses RabbitMQ:

• The volume (~3,000 orders/day, a few events per order) is orders of magnitude below the threshold where Kafka justifies its operational complexity. With ~25 engineers, the Platform team cannot dedicate anyone to looking after partitions.
• Topic routing with one queue per consumer directly models the context map from 02-03: each service subscribes to what it cares about and nobody else finds out.
• The DLQ, prefetch and confirms cover the guarantees the saga needs without extra code.
• In 02-05 we ruled out event sourcing "for now"; if it is ever adopted, Kafka's retention and replay would be the reason to migrate, and the standard event envelope will make that migration less painful.
• Marta ruled out managed queues so as not to tie the system to one cloud at a time when it is still being decided where Kubernetes will run.

There is one topic this lesson has brushed against in every snippet: the shape of the payload of each event (which fields order.created carries, what happens when one has to be added or another changed) and that version field in the envelope. That is the events' contract, and it is covered together with the APIs' contract in 03-06.

• Publishing directly to a queue (sendToQueue) instead of to the exchange. It works until a second service needs the same event; then the producer has to be touched. With the exchange you just add a binding.
• ack before processing "so it goes fast". It turns at-least-once into at-most-once: a restart halfway through and the order is left without a reservation forever.
• Requeuing without a limit (nack(msg, false, true) in every catch). A poison message monopolizes the queue. One redelivery and then to the DLQ.
• Forgetting prefetch. The first replica to start takes all the pending messages; the others just watch.
• Non-durable queue or non-persistent message. Everything looks fine until the first broker restart. Both flags, always, for business events.
• Event payloads that point to data ({"orderId": "ord-88213"} and have the consumer call GET /orders/ord-88213). It reintroduces the temporal coupling we wanted to remove. The event carries what the consumer needs (that is why order.created includes lines, prices and email).
• Relying on ordering across queues or across replicas. Design consumers to tolerate payment.confirmed before their own DB reflects stock.reserved (the state machine from 02-05 and processOnce help).
• Ignoring the DLQ. Without an alert on *.dlq, messages pile up for months. The alert is defined in 06-05; starting today, look at it in the RabbitMQ console.
• One connection per request. AMQP connections are expensive. One connection per process, one channel per consumer or publisher, reused.

Exercise 1. The Payments & Communications team wants Notifications to also send a "we have received your order" email as soon as it is created (in addition to the confirmation one). Say which binding has to be added, to which queue, and what does not have to be touched. Then reason: would it make sense for Notifications to use the same notifications.orders queue or a new notifications.orders-received one? Give one argument in favor of each option.

Exercise 2. A Payments consumer processes stock.reserved, calls the external payment provider, which charges Ana €79.70, and right before the ack the instance dies. RabbitMQ redelivers the message to another replica. Explain step by step what happens with and without processOnce, and which additional guarantee the Payments consumer needs with respect to the payment provider (hint: 03-01 talked about it under another name).

Exercise 3. Write a function moveFromDlqToQueue(channel, queueName, max) with amqplib that reads up to max messages from ${queueName}.dlq with channel.get() (synchronous fetch, no subscription) and republishes them to the techcorp.events exchange with the original routing key (available in msg.fields.routingKey or, after dead-lettering, in the x-death header), keeping persistent: true, and acks each one in the DLQ only after republishing it. Comment every line.

Solution 1.

It is enough to add the order.created binding to the Notifications queue (bindQueue('notifications.orders', 'techcorp.events', 'order.created')) and register an 'order.created' handler in its consumer. Orders does not have to be touched (it keeps publishing exactly the same), nor Inventory, nor the exchange: that is what topic pub-sub buys you. The order.created payload already carries customer.email and customer.name precisely so that Notifications does not have to call anyone.

Same queue: simpler, one consumer, one prefetch, one DLQ to watch; the relative order "received → confirmed" for the same order is better preserved. New queue: it isolates failure (if the "received" email breaks and fills the DLQ, the confirmation ones keep going out) and allows scaling and prioritizing separately (the confirmation ones are more important). For TechCorp, at the current volume, the same queue; split them when there is an operational reason.

Solution 2.

Without processOnce: the second replica receives the same stock.reserved (same eventId), does not know it was already processed, calls the payment provider again and charges €79.70 twice; it also publishes two payment.confirmed. With processOnce: the second replica looks up processed_events for (evt-..., 'payments')... and here is the catch: if the first replica died before committing the transaction that inserts into processed_events, the event is not recorded as processed, and the second replica will charge too. processOnce protects against "I processed and died before the ack" only if the effect and the record in processed_events are in the same local transaction, and the call to the external payment provider cannot be inside that transaction.

The additional guarantee: the payment provider must accept an idempotency key (most real providers do), and Payments must use as the key something derived from the order (ord-88213) or from the eventId, exactly the same concept as the Idempotency-Key header from 03-01, but from Payments outward. That way, the second call to the provider returns the same charge instead of creating another one. General rule: idempotency is needed at every boundary where there is a non-reversible effect.

Solution 3.

Notes: we fill in msg.properties.type in publishEvent with envelope.type, so it serves as a fallback if x-death were missing. With a confirm channel it would be even safer to wait for waitForConfirms() before the ack. And yes, this script can duplicate an event in the worst case; as with everything in this lesson, consumer idempotency is what makes it harmless.

Asynchronous messaging is what makes the saga from 02-05 possible: it removes the temporal coupling between services, absorbs peaks and lets a single fact (order.confirmed) reach several interested parties without the producer knowing about them. We have fixed the vocabulary (event versus command, queue versus pub-sub, ack/nack, redelivery, DLQ), accepted at-least-once + idempotency as the realistic guarantee, and built TechCorp's topology in RabbitMQ: a topic exchange techcorp.events, routing keys equal to the event type, one durable queue per consumer (inventory.orders, payments.stock, notifications.orders, orders.saga, orders.customers) with its DLQ, and the amqplib code to publish the standard envelope with persistent: true and to consume with prefetch, ack after processing and nack to the DLQ. With REST for the synchronous side and RabbitMQ for the asynchronous side, TechCorp now has its two main channels.

But REST/JSON is neither the only form of synchronous call nor always the best: when two internal services talk to each other thousands of times a minute, JSON's verbosity and the lack of a typed contract weigh heavily, and when a front end needs to compose data from several services, REST forces many requests or huge responses. For the first case there is gRPC; for the second, GraphQL. In the next lesson we will look at both, with their Node.js code, and decide where they fit (and where they do not) at TechCorp.