The previous lesson ended with three awkward questions. If inventory deducts the stock and dies before acknowledging the message, the broker redelivers it and the stock is deducted twice. If orders saves the order in PostgreSQL and crashes before publishing order.created, inventory and analytics will never know that the order exists. And if a malformed message makes the consumer fail over and over again, all the messages queued behind it are left waiting. None of the three is a rare case: they are direct consequences of the fallacies from 01-04 applied to messaging, and they will show up in production during the very first campaign.

This lesson presents the patterns with which the industry has learnt to live with them. First we will put a name to what a broker can and cannot guarantee (at-most-once, at-least-once and the myth of exactly-once), and see how those guarantees depend on where the ack is placed. Then we will finally close the duplicates problem that we left open in 01-04 and 02-02, with idempotent consumers backed by a table in PostgreSQL. We will solve the dual write with the transactional outbox pattern and a relay, implement request/reply over messaging, scale out with competing consumers while respecting per-partition ordering, and deal with poison messages using retries and dead letter queues. We will finish with event schema versioning and a table that maps each pattern to the problem it solves. Sagas (transactions across services) and the circuit breaker (resilience of synchronous calls) are left for 03-05 and 07-04.

1. Delivery guarantees: at-most-once, at-least-once and the myth of exactly-once
2. Idempotent consumers: closing the duplicates problem
3. The dual write problem and the transactional outbox pattern
4. Request/reply over messaging
5. Competing consumers and per-partition ordering
6. Poison messages: retries with backoff and dead letter queues
7. Event-driven and event sourcing: two different things
8. Event schemas and versioning
9. Pattern table: which problem each one solves
10. Common mistakes and tips
11. Exercises
12. Conclusion

In 02-02 we saw the invocation semantics of RPC. The delivery guarantees of messaging are the same idea with a broker in the middle, and the factor that decides them is when you acknowledge (ack in RabbitMQ, offset commit in Kafka) relative to when you process:

The myth, stated precisely. Kafka offers a feature called exactly-once semantics (idempotent producers and Kafka transactions) which guarantees that a message is not duplicated inside Kafka: if the producer retries because of a timeout, the broker deduplicates; and an application that reads from one topic and writes to another can do so atomically. It is valuable for Kafka → Kafka pipelines (Module 5). But as soon as the effect of the message leaves Kafka (an UPDATE in PostgreSQL, an email, a charge), the window between "effect applied" and "offset committed" reappears, and nobody can close it from outside. The same goes for MQTT's QoS 2, which guarantees a single delivery to the client, not a single effect. The practical conclusion is the one from 02-02: at-least-once in the transport, idempotency in the consumer. Everything else in this lesson is built on that foundation.

An operation is idempotent if running it N times produces the same state as running it once. Some are idempotent by nature (SET stock = 118, "mark the order as paid", "insert with a primary key"); others are not (stock = stock - 2, "send an email", "charge €14.50"). For those that are not, the technique is to have the consumer remember which messages it has already processed and discard the repeats. Two requirements:

1. An idempotency key per message: a unique identifier generated by the producer and stable across retries. It is the event_id of the envelope from 02-04, and it was the reservation_id of the gRPC request in 02-03. Without it there is no way of knowing that two messages are "the same one".
2. Record the key and apply the effect in the same transaction. If the key is recorded in one transaction and the effect applied in another, the window reappears: the key is recorded, the process dies, the effect never happens, and the retry is discarded as a duplicate (loss). If it is the other way round, the effect is applied, the process dies, and the retry applies it again (duplicate). With both in the same transaction, either both happen or neither does.

inventory already has its own database (one of the principles of 01-06), so we add the table there:

The composite primary key allows two different pieces of logic inside inventory (deducting stock and, say, notifying the producer) to each process the same event once. The Kafka consumer from 02-04, now idempotent, with psycopg (version 3):

Let's walk through the possible failures with this code:

• It dies after the PostgreSQL COMMIT and before the Kafka commit. Kafka redelivers the message; the INSERT collides with the primary key; ON CONFLICT DO NOTHING does not insert; fetchone() returns None; the answer is duplicate without touching the stock; the offset is committed. No duplication.
• It dies in the middle of the transaction. PostgreSQL does a ROLLBACK: neither the key nor the deduction persists. Kafka redelivers; it is processed from scratch. No loss.
• It dies before starting. Kafka redelivers. Trivial.
• The producer published the same event twice (it retried because of a broker timeout). Same event_id, same outcome: the second copy is a duplicate. This is the situation from the simulation in 01-04, now solved: the retry no longer turns the loss of a message into duplicated effects.

The same applies to the synchronous call from 02-03: ReserveStock can record reservation_id in processed_messages with consumer = 'inventory.grpc_reservation' and return the stored response if the call is repeated, so the "unknown status" after a DEADLINE_EXCEEDED stops being a problem: orders can retry with the same reservation_id without risk. It is the at-most-once semantics of 02-02 built with a table. Two operational considerations: the table grows (one row per message) and must be purged by age (beyond the topic's retention period no retry can arrive any more: DELETE ... WHERE processed_at < now() - interval '14 days'); and if the consumer has no transactional database (for example, it sends emails), idempotency has to rely on the target system (an email provider that accepts an idempotency key) or settle for at-most-once.

Now for the producer's side. In 02-04, orders did two things when creating an order: write it to its PostgreSQL database and publish order.created to Kafka. These are two different systems and there is no transaction that spans both (this is the dual write):

• If it writes to the database and crashes before publishing, the order exists but nobody finds out: inventory does not deduct, analytics does not count, delivery does not deliver.
• If it publishes first and then the COMMIT fails, everyone reacts to an order that does not exist.
• If it publishes inside the transaction "so that it gets rolled back", it does not get rolled back: Kafka takes no part in PostgreSQL's ROLLBACK.

The transactional outbox pattern solves the problem by turning two writes into one: the event is written to a table in the same database, in the same transaction as the order. A separate process, the relay, reads that table and publishes to Kafka.

The table, in the orders database:

The write in orders:

And the relay, an independent process that can run as several instances thanks to FOR UPDATE SKIP LOCKED (each one locks different rows):

Let's examine the relay under the same magnifying glass as the consumer. If it dies after flush() and before the UPDATE, the transaction is rolled back, the rows are still pending and on the next pass they are published again: the relay is at-least-once, and it produces duplicates with the same event_id (the id of the outbox row). Those duplicates are absorbed by the idempotent consumer from section 2. The two patterns need each other: the outbox guarantees that no event is lost; idempotency guarantees that none is applied twice. And ORDER BY created_at together with the aggregate_id partition key preserves the order of each order's events.

CDC as an alternative to the relay. Instead of a process that polls the table, Change Data Capture (CDC) tools such as Debezium read PostgreSQL's write-ahead log and publish every insert into outbox to Kafka with millisecond latency and no repeated queries. It is the recommended outbox implementation at scale; the polling relay is easier to understand and good enough to start with. We only mention it: the mechanics of the replication log belong to 03-04.

Sometimes you want the temporal decoupling of messaging but you need an answer: an internal service that asks payments to authorise a card and wants the result, even if it takes seconds. The pattern uses two properties of the AMQP message:

• reply_to: the name of the queue where the requester waits for the reply (often an exclusive, temporary queue that it has created itself).
• correlation_id: an identifier that the requester puts in the request and the server copies into the reply, so that the requester can match replies to requests when it has several in flight.

On the payments side, the consumer processes the request and publishes the reply to props.reply_to with correlation_id=props.correlation_id. (With BlockingConnection, process_data_events(time_limit=...) is the way to handle the replies that arrive while waiting; in an asynchronous client the loop would be different, but the idea is the same.) This pattern keeps the advantages of messaging (if payments is overloaded, requests wait in the queue instead of overloading it further; if it restarts, they are not lost) in exchange for more complexity than a gRPC call. At Kilometre Zero it will be used sparingly: for quick queries, gRPC (02-03) is simpler; for consequences, plain events. Its place is slow or spiky operations where you want an answer but do not want to block the server.

When inventory cannot keep up during "Artisan Cheese Week", the solution is to start more instances that share the queue (RabbitMQ) or the group (Kafka): competing consumers. Each message is processed by a single instance, and throughput scales with the number of instances... up to a limit, which is different for each broker:

• In RabbitMQ, any number of instances can compete for the same queue, but spreading messages among them means ordering is lost: Anna's order.created may go to instance 1 and order.cancelled to instance 2, which may process it first. If ordering matters, you have to serialise some other way (one queue per key, or the consistent hash exchange plugin).
• In Kafka, the limit is the number of partitions (02-04) and ordering is guaranteed within each partition: all the events for P-2026-000123 are processed by the same instance in order, while those of other orders are spread around. This is the main reason why Kilometre Zero's domain events go to Kafka.

The price of per-partition ordering is head-of-line blocking of the whole partition (the same phenomenon as in TCP, 02-01): if the message at offset 41 of partition 2 takes ten seconds, offsets 42 onwards in that partition wait, even if they belong to other orders. Hence the importance of the consumer not getting stuck on one message, which is the subject of the next section.

A poison message is one that makes the consumer fail every time it tries: malformed JSON, a product that does not exist in stock, a bug in the consumer triggered by certain data. With at-least-once and nothing else, the broker redelivers it indefinitely and the consumer is stuck in a loop, blocking its partition or its queue. Two classes of failure need to be distinguished:

• Transient: the database is not responding, a remote service times out. Retrying after a wait makes sense, and the wait should grow (exponential backoff: 1 s, 2 s, 4 s, 8 s...) so as not to hammer a sick system.
• Permanent: invalid data, a business rule violation (InsufficientStock from section 2), a bug. Retrying changes nothing; the message must be set aside so that the rest can move on, and someone has to look at it.

The destination for messages that are set aside is the dead letter queue (DLQ). In RabbitMQ it is native: you declare the queue with a dead-letter exchange and, when a message is rejected without requeueing, the broker moves it there.

(An unbounded requeue=True can degenerate into the loop; in RabbitMQ the practice is to count attempts in a header, x-death, and to use a wait queue with a TTL for the backoff. The idea is the same as the one we are about to implement in Kafka.)

Kafka has no native DLQ: it is built with retry topics and a dead letter topic, and the consumer decides which one to send each failure to:

A consumer of the .retry topic reads each message, sleeps until not_before if necessary, and processes it with the same function. Considerations that are not visible in the code: the message that goes to .retry loses its place in the order of the original partition (it will be processed after other, more recent ones), which is acceptable for a stock deduction and may not be for a created → cancelled sequence; the DLQ needs alerts (07-01) and a procedure for examining, fixing and republishing; and InsufficientStock from section 2 is a good example of a permanent error whose resolution is not technical but a business matter (notify the customer, cancel the order), which is the territory of the sagas of 03-05. Retries, backoff and circuit breakers for synchronous calls are covered in 07-04; here we have only seen those that belong to message consumption.

Everything we have done in 02-04 and in this lesson is event-driven architecture: services communicate facts that have happened (order.created) and others react. Each service's state still lives in its tables (orders, stock); the events are notifications.

Event sourcing is something else, which is often confused with the above: it means that the state is not stored; the events are, and the state is rebuilt by replaying them. Anna's order would not be a row with status = 'paid', but the sequence OrderCreated, LineAdded, OrderPaid, and the row would be a derived projection. It gives you a complete audit trail and lets you rebuild any past state, at the cost of considerable complexity (projections, snapshots, evolution of historical events). You can be event-driven without event sourcing (Kilometre Zero is) and vice versa. We mention it so that the terms do not get mixed up; developing it is beyond the scope of this course.

An event is a contract between the producer and all present and future consumers, including those that will read the Kafka history six days from now. The schema evolution rules of 02-03 apply to it even more strictly, because you cannot know how many consumers there are or force them to upgrade. The standard envelope we have been using already provides the mechanism:

The rules, adapted from 02-03:

• Compatible changes (do not increment version): adding optional fields to data; adding new event types. Consumers must ignore unknown fields and tolerate the absence of the new ones.
• Incompatible changes (increment version): removing or renaming a field, changing its type or its meaning. During the transition, the producer can publish both versions or the consumers can accept both (if event["version"] == 1: ...). The meaning of a field is never changed while keeping its name.
• Schema Registry: with Avro or protobuf on Kafka, a central service stores every version of each topic's schema, validates on the producer side that a change is compatible (backward, forward or both, depending on the policy) and lets consumers deserialize old messages with the schema they were written with. It is the way to turn the rules above into an automatic check, and the natural next step when JSON falls short. Kilometre Zero starts with JSON and the envelope; the schema registry is introduced with the data pipeline of Module 5.

• Looking for exactly-once in the broker configuration. It is not there. It is in the idempotent consumer, and there is no shortcut.
• Recording the idempotency key outside the effect's transaction. It reopens the window you were trying to close. Same transaction, always.
• Using something that changes between retries as the idempotency key. If the producer generates a new event_id on every retry, the consumer cannot recognise the duplicate. The key is generated once and reused.
• Publishing to Kafka "inside" the PostgreSQL transaction. The ROLLBACK does not undo it. Outbox or nothing.
• A relay that marks rows as published before the flush. If Kafka does not confirm, the event is lost for ever with the row marked. Confirmation first, mark afterwards.
• Retrying permanent errors. Malformed JSON does not fix itself by waiting 4 seconds; it blocks the partition three more times. Classify your errors and send the permanent ones to the DLQ at the first attempt.
• A DLQ with no alerts and no owner. It is a black hole where orders silently disappear. Every DLQ needs an alert, a person and a procedure.
• Changing the meaning of a field without changing the version. A consumer that re-reads the history will read data with two meanings under the same name. New field or new version.
• Tip: test idempotency explicitly: in the test environment, publish every event twice and check that the final state is the same. It is the cheapest test there is and the one that prevents the most incidents.
• Tip: measure the lag of the relay (outbox rows with published_at IS NULL and their age) and of each consumer group. They are the two indicators that tell you whether asynchrony is working or piling up invisible debt.

This payments consumer processes order.created and runs the charge against the external gateway. Identify all the delivery-guarantee problems it has, say what can go wrong in each case (loss, duplicate, blocking) and rewrite it applying the patterns from the lesson. Assume that the gateway accepts an Idempotency-Key header and that payments has its own PostgreSQL database.

inventory must publish stock.updated (with product, before, after, reason) every time the stock changes, whether through the gRPC reservation from 02-03 or through the consumption of order.created in section 2. Design the outbox table for inventory, modify process_order_created to write the event in the same transaction, and explain what the combination (idempotency + outbox) guarantees in the face of each of these failures: (a) the consumer dies between the COMMIT and the offset commit; (b) the relay dies after publishing and before marking; (c) Kafka is unavailable for 10 minutes.

For the delivery consumer that assigns a courier to each order.paid, classify each of these failures as transient or permanent, say where the message goes (retry with backoff, DLQ) and what the operator does in each case: (1) the external maps service returns 503; (2) the order has a delivery address in a city where Kilometre Zero does not deliver; (3) KeyError: 'delivery_address' on a version: 1 event; (4) the delivery database refuses the connection during a restart; (5) there is no free courier in Tarragona right now.

Solution 1:

Problems: (1) enable.auto.commit=True commits offsets periodically regardless of whether the message has been processed: if the process dies after the auto-commit and before charging, the charge is lost (at-most-once); if it dies after charging and before the auto-commit, the customer is charged twice (at-least-once without idempotency). (2) The charge to the gateway and the INSERT are two writes with no common transaction: if the gateway charges and the INSERT fails, there is a charge with no record; on retry, another charge. (3) There is no idempotency key either towards the gateway or in the database: the same order.created delivered twice charges Anna twice. (4) msg.error() is silently ignored. (5) Any exception (malformed JSON, invalid card) kills the loop or, if it were caught, would block the partition. (6) It does not publish payment.confirmed, so nobody finds out about the charge (a dual write still to be dealt with). Rewrite:

One unavoidable window remains: if the process dies between gateway.charge and the COMMIT, the transaction is rolled back (no key, no record), the message is retried, and the second charge with the same Idempotency-Key makes the gateway return the charge already made instead of repeating it. The external system's idempotency closes what the local transaction cannot cover. Without that header, we would have to record "charge started" before calling and reconcile afterwards: that is the situation in which the sagas of 03-05 become necessary.

Solution 2:

With aggregate = 'product' and aggregate_id = product, so that the changes to a given product go in order to the same partition of inventory.events. In process_order_created, inside the same with conn.transaction(), after each successful UPDATE:

(And the same in ReserveStock on the gRPC server, which would stop using the in-memory dictionary and use the same database instead.) Guarantees: (a) the consumer dies between the COMMIT and the offset commit: the stock has been deducted and the event is in outbox; the relay will publish it; the redelivered message is detected as a duplicate and generates neither a deduction nor a second event. (b) The relay dies after publishing and before marking: the event is published twice with the same event_id; catalog and analytics, if they are idempotent, ignore it the second time; if catalog only does SET indicator = (after < 5), it is idempotent by nature and does not even need the table. (c) Kafka down for 10 minutes: inventory carries on processing (if it reads from RabbitMQ) or stops (if it reads from Kafka), but in both cases it loses nothing: the events pile up in outbox with published_at IS NULL and the relay publishes them in order when Kafka comes back. It is exactly the behaviour asked for in solution 3.3 of lesson 01-06, now implemented.

Solution 3:

1. Transient: 503 means "come back later". Retry with backoff (1, 2, 4 s...) up to the maximum; after that, DLQ. The operator checks the status of the maps provider; once it recovers, they republish the DLQ.
2. Permanent, business-related: no retry will fix it. DLQ with reason city_not_covered, and probably a delivery.rejected event so that orders cancels and payments refunds (a saga, 03-05). The operator or, better still, validation in orders when the order is created, should have prevented it earlier: the DLQ reveals a gap in the validation.
3. Permanent, contract-related: the consumer assumes the version: 2 schema and receives a version: 1 one (probably from the history or from an orders that has not yet been upgraded). DLQ with reason schema, an alert to the team, and the fix goes in the consumer (accept both versions, section 8); once it is deployed, the messages in the DLQ are republished. It is not a data failure but that of a consumer that did not follow the compatibility rules.
4. Transient: database restart. Retry with backoff. Here it is especially important not to commit the offset without first having moved the message to .retry; with the database down, idempotency cannot even be recorded, so the message must remain intact to be retried.
5. Transient, but business-related and long-lasting: it may take 20 minutes for a courier to become free. A backoff measured in seconds does not fit; it is better for the consumer to process the message by recording the order as "pending assignment" in its database (idempotently) and commit, and for a periodic process to try to assign the pending ones. Turning a long wait into persisted state is preferable to keeping messages bouncing between retry topics, which would also lose their ordering relative to a possible later order.cancelled.

This lesson has closed the module by solving the problems that the previous ones kept leaving open. Delivery guarantees depend on where you acknowledge relative to where you process: acknowledging before loses messages (at-most-once), acknowledging after duplicates them (at-least-once), and exactly-once does not exist end to end, so the strategy is at-least-once in the transport and idempotency in the consumer. The idempotent inventory consumer, with its processed_messages table in the same transaction as the effect, has closed the duplicates problem we had been carrying since the simulation in 01-04 and the semantics of 02-02, and the same technique makes retries of the gRPC call from 02-03 safe. The outbox pattern, with its FOR UPDATE SKIP LOCKED relay, has eliminated the dual write in orders: state and event are saved together or not at all. We have also seen request/reply over queues with reply_to and correlation_id, competing consumers with per-partition ordering, retries with backoff, dead letter queues with their alerts and their procedure, the difference between event-driven and event sourcing, and event versioning with the standard envelope and the rules inherited from 02-03. The table in section 9 summarises which pattern solves which problem and where Kilometre Zero uses it.

With this, Module 2 has kept its promise: orders and inventory are two processes that talk to each other reliably, synchronously over gRPC when an answer is needed and asynchronously over Kafka for the consequences, and analytics listens without anyone having to wait for it. But look at what has happened along the way: inventory now has its own stock table and orders its own orders table, and the truth about Montblanc Dairy's last aged cheese is no longer in a single place. When Anna reserves it and catalog receives the stock.updated event half a second later, for that half second Mark sees a cheese in the catalogue that no longer exists. When inventory has the inv-bcn and inv-vlc replicas, which of the two is right if they differ? What exactly does "consistent" mean when the data lives in several places, and what can be promised to a customer when the network between them fails? These are the questions of Module 3, Consistency and Replication, which begins with consistency models: the precise vocabulary for saying what a distributed system guarantees about its data and what it does not.