In the previous lesson we saw what an event is and who produces or consumes it. Now we go one level down: how does that message physically travel from one service to another reliably when the two do not know each other and may be starting up, going down, or overloaded at different times? The answer is asynchronous messaging through message brokers. A broker is an intermediate piece of infrastructure that receives messages from producers, stores them durably, and delivers them to consumers when the latter are ready.

Mastering asynchronous messaging is essential because the reliability and consistency of any distributed system depend on it. In this lesson we will study the two basic primitives (queues and topics), delivery guarantees, the problem of duplicates and how to solve it with idempotency, and we will compare the three most widely used technologies on the market: RabbitMQ, Apache Kafka, and Amazon SQS.

1. Why asynchronous messaging?
2. Queues (point-to-point) vs Topics (publish/subscribe)
3. Delivery guarantees: at-most-once, at-least-once, exactly-once
4. The problem of duplicates and idempotency
5. Comparison: RabbitMQ vs Kafka vs Amazon SQS
6. Practical example: producer and idempotent consumer
7. Common mistakes and tips
8. Exercises and solutions
9. Conclusion

In a synchronous call, if service B is down, A's call fails. With asynchronous messaging, A deposits the message in the broker and keeps working; when B recovers, it will consume it. This provides:

• Temporal decoupling: producer and consumer do not need to be alive at the same time.
• Peak buffering: if 10,000 messages arrive at once, the broker holds them and the consumer processes them at its own pace.
• Resilience: a slow or down consumer does not bring down the producer.

There are two message distribution models.

A message in a queue is processed by a single consumer. If there are several consumers connected to the same queue, the broker distributes (balances) the messages among them, but each message goes to only one. It is used to distribute work (work queue).

A message published to a topic is delivered to all subscribers. It is used to broadcast events to multiple interested parties.

When a message travels over the network, acknowledgments (acks) can be lost, processes can crash, etc. That is why different levels of guarantee exist:

Key points:

• At-most-once: the consumer acknowledges before processing. If it fails, the message is lost. Only valid if losing some data is acceptable (e.g., non-critical metrics).
• At-least-once: the consumer acknowledges after processing successfully. If it fails just before the ack, the message is redelivered → there may be duplicates. It is the most common and recommended level.
• Exactly-once: semantically perfect, but costly. In pure distributed systems it is very hard to guarantee end to end; it is usually achieved by combining at-least-once with idempotency in the consumer (we will see this next). Kafka offers "exactly-once" within its own boundaries (transactions), but as soon as the effect leaves Kafka (writing to another DB, calling an API), it again depends on idempotency.

Real-world golden rule: design for at-least-once and make your consumers idempotent. In practice, that gives you the effect of exactly-once.

An operation is idempotent if executing it several times produces the same result as executing it once. If our consumers are idempotent, at-least-once duplicates stop being a problem.

Common techniques to achieve idempotency:

1. Idempotency key / message ID: record the IDs already processed and discard repeats.
2. Naturally idempotent operations: UPDATE balance SET value = 100 (absolute assignment) is idempotent; UPDATE balance SET value = value + 100 (increment) is not.
3. UPSERT with a unique key: insert or ignore if it already exists.

Explanation:

• message_id is the primary key: the database guarantees there are no two identical ones.
• ON CONFLICT ... DO NOTHING makes the second attempt to insert the same ID produce neither an error nor an effect. This way we detect the duplicate without complex additional logic.
• If the INSERT affected 0 rows, it was a duplicate and we can skip processing.

Practical summary:

• RabbitMQ: excellent when you need flexible routing (exchanges with rules) and traditional work-queue patterns.
• Kafka: the choice for high volume, retention, and replay of events; the foundation of event sourcing and streaming (lesson 05-05).
• SQS: the simplest option if you are already on AWS and just want to decouple without managing servers.

Let's look at a Kafka consumer in Spring that applies at-least-once + idempotency.

Detailed explanation:

• @KafkaListener subscribes the method to the topic payments.confirmed. The groupId "accounting" identifies this consumer group; Kafka distributes the partitions among the group members.
• registerIfNew(...) attempts to insert the ID (as in the SQL above). It returns false if it already existed → it is a duplicate, we acknowledge and exit without reprocessing.
• The acknowledgment (ack.acknowledge()) is done after postEntry. If the process dies before the ack, Kafka will redeliver the message (at-least-once), but idempotency will prevent the double entry.

• enable-auto-commit: false is essential: if Kafka acknowledged on its own, it could acknowledge before we finished processing, and we would lose messages upon a failure.
• ack-mode: manual delegates to our code the exact moment of acknowledgment.

• Acknowledging before processing. It turns your at-least-once into an accidental at-most-once and you lose messages upon failures. Always acknowledge at the end.
• Believing that "exactly-once" eliminates the need for idempotency. As soon as the effect crosses the broker's boundary (another DB, an external API), you need idempotency all the same.
• Not sizing the dead-letter queue (DLQ). Messages that fail repeatedly must go to a discard queue so they do not block the main queue in an infinite loop of retries.
• Using a topic when you wanted a queue (or vice versa). Broadcasting a command to "everyone" can execute the same action N times.
• Tip: always define a unique message_id field in your events from day one; adding it later is painful.

1. A team processes payments with a consumer that acknowledges the message right upon receiving it and then contacts the banking gateway. If the process dies between the ack and the banking call, what real guarantee do they have and what problem occurs? How would you fix it?
2. Indicate for each case whether you would use a queue or a topic: (a) send the welcome email exactly once; (b) notify inventory, billing, and CRM of a new order; (c) distribute 1,000 PDF-generation tasks among 5 workers.
3. Write an idempotent SQL statement to "mark an order as paid" that can be executed several times without side effects.

1. They have at-most-once: if it dies after the ack, the message is not redelivered and the payment never reaches the gateway (it is lost). Fix: acknowledge after the banking call (at-least-once) and make the operation idempotent using the payment's idempotency key so as not to charge twice upon a retry.
2. (a) Queue (a single receiver, a single time). (b) Topic (three subscribers receive the event). (c) Queue (work distribution among workers).
3. For example:

Running it again changes nothing because the condition status <> 'PAID' is no longer met and paid_at is preserved with COALESCE.

Asynchronous messaging is the circulatory system of event-driven architectures. We distinguished queues (one receiver, load distribution) from topics (all subscribers, broadcast). We understood the three delivery guarantees and why at-least-once + idempotency is the pragmatic combination the industry uses. Finally, we compared RabbitMQ, Kafka, and SQS to know when to choose each one.

In the next lesson, "Event Patterns: Event Sourcing and CQRS", we will see how, instead of storing only the current state, we can store the complete sequence of events as the source of truth, and how to separate the read and write models to scale.