The four previous lessons have given MercadoFresco the pieces: queues to decouple, topics to hand out, a bus to route by content and workflows to orchestrate. But they have left questions postponed with a "we will see that in 07-05", and they all point in the same direction. What exactly does "at least once" mean when the thing being duplicated is a charge of 48.20 €? How do you write a consumer that can receive the same message three times without charging three times? How many times should you retry, and when should you stop so as not to knock over the service that is recovering? What exactly do you do with 340 messages in a DLQ on a Monday morning? And how do you guarantee that an order written to Aurora always gets published, if there is no transaction spanning the database and the event bus?

This lesson introduces no new services. It turns the four you already know into design criteria: when to use each one, what guarantees they really give and what you have to add yourself because the service does not. It is the lesson that separates an architecture that works in the demo from one that survives a Friday in October with the payment provider running at half speed.

Cost warning. The mercadofresco-idempotencia table on demand with a 24 h TTL costs around 0.90 USD a month at MercadoFresco's volume. That is not the expensive part: a duplicate charge is, because on top of the 48 € it costs a call to customer service and a refund. Data is fictitious.

1. Delivery guarantees: the three of them, and why one is almost a lie
2. Idempotency: the property that fixes everything
3. The mercadofresco-idempotencia table
4. A complete idempotent consumer
5. Retries: exponential backoff and jitter
6. Which errors deserve a retry and which do not
7. Where to retry: SDK, service or workflow
8. The retry storm
9. A circuit breaker for the payment provider
10. Dead-letter queues: classifying and reprocessing
11. Marta's DLQ runbook
12. Order and grouping: when it really matters
13. The outbox pattern and the dual-write problem
14. Backpressure, buffering and throttling
15. Decision table: SQS, SNS, EventBridge, Step Functions or synchronous
16. MercadoFresco's complete integration architecture
17. Common mistakes and tips
18. Exercises
19. Conclusion

Practically all serious messaging chooses at least once, for a simple reason: between losing an order and processing it twice, the second has a fix and the first does not.

"Exactly once" almost never exists end to end, and it is worth understanding why. Picture the consumer that charges the card:

1. It receives the message from cola-mercadofresco-pedidos.
2. It calls the gateway. The gateway charges.
3. The network drops before the response arrives.
4. The process dies, or throws an exception, and does not delete the message.
5. The visibility timeout expires. Another consumer receives the same message.
6. It calls the gateway. The gateway charges again.

No messaging service can prevent this, because the problem is not in the delivery: it is that charging and confirming that you have charged are two different operations and a failure fits between them. FIFO SQS deduplicates the producer's send for 5 minutes, not the consumer's effect. Standard Step Functions guarantees that its own machine does not repeat a step, but if the step's Lambda charged and then failed to respond, Step Functions will retry the step.

The operational conclusion is short and has to be accepted without resistance: the system delivers at least once; the exactly-once effect is put there by your consumer. And that is called idempotency.

An operation is idempotent if running it several times with the same input produces the same result as running it once. It does not mean "it does nothing the second time": it means that the final state is the same.

The rule that follows: operations that set a value are idempotent; those that modify it relatively or produce external effects are not. And when an operation is not idempotent by nature, it has to be made idempotent by adding an idempotency key.

An idempotency key is a stable identifier of the logical operation, not of the message. The difference is crucial:

• ❌ The SQS MessageId. It changes if the producer resends the same work. Useless.
• ❌ A UUID generated in the consumer. It changes on every attempt. Useless for anything.
• ✅ f"cobro:{pedido_id}" — derived from the domain, stable across retries and across producers.
• ✅ f"correo-confirmacion:{pedido_id}" — one email per order, whatever the path taken.
• ✅ f"reserva:{pedido_id}:{sku}" — one stock movement per order line.
• ✅ A hash of the canonical content, when there is no natural identifier.

Notice that the key includes which operation as well as on what. PED-084417 on its own will not do: the same order generates a charge, an email and several reservations, and those are different operations, each of which must be able to run once.

Many third-party APIs accept the key directly. MercadoFresco's payment gateway supports an Idempotency-Key header: sending cobro:PED-2026-084417, a second attempt returns the same charge instead of creating a new one. When the provider supports it, that is always the best option, because the guarantee comes from whoever holds the state. When it does not, we have to keep the record ourselves.

An item looks like this:

Four design decisions, all with a reason:

The estado has three values, not two. EN_CURSO, COMPLETADO and FALLIDO. EN_CURSO is the one that solves the hard case: two consumers that receive the same message at the same time. The first marks EN_CURSO with a conditional write; the second sees that it exists and backs off. With only two states, both would see "it is not there" and both would charge.

The resultado is stored. If the work has already been done, the second attempt must not simply ignore the message: it must return the same result as the first time. In a state machine, that means the step carries on with the same referencia_cobro and the process moves forward instead of breaking.

A 24-hour TTL (expira_en in epoch seconds, the same mechanism as mercadofresco-carritos in 06-02). The window must comfortably cover the maximum life of a message in the system: the queue's retention (4 days) is too much, and one hour is too little if something sits in the DLQ and gets reprocessed in the afternoon. 24 hours is MercadoFresco's balance. Without a TTL, the table grows indefinitely and you pay storage for records nobody is ever going to query.

Encrypted with alias/mercadofresco-datos, because the resultado may contain charge references.

The four points that make this really work:

1. The conditional write is atomic. DynamoDB guarantees that only one consumer wins the race. No distributed lock is needed; it is a single call.
2. The EN_CURSO expires. Without the condition on caduca_bloqueo, a consumer dying between the put_item and the complete would leave the key locked for 24 hours and that order never charged.
3. A transient failure releases; a permanent one does not. If the network failed, the retry has to be allowed. If the card was declined, retrying is throwing requests in the bin.
4. The key travels to the gateway too. If the provider supports Idempotency-Key, then both nets are in use: ours and theirs. If ours fails right in the gap between charging and recording, theirs covers us.

Powertools for AWS Lambda implements this complete pattern with a decorator, including the table, the TTL, the lock and the caching of the result. In Python it is enough to put @idempotent on the handler, configuring DynamoDBPersistenceLayer and the event_key_jmespath that extracts the key from the event. In production that is the sensible thing; writing it by hand once, as here, is what lets you understand what it does and debug it when it fails.

Retrying without waiting is counterproductive: if the service is failing because it is saturated, immediate retries saturate it further. Exponential backoff multiplies the wait on each attempt; jitter randomises it.

Jitter is not a tuning detail: it is what stops the recovery from causing the next outage. Without it, if the gateway goes down for 30 seconds, the 450 executions in flight retry at exactly the same instant and knock it over again just as it was coming back up.

random.uniform(0, ...) is full jitter, the variant that spreads the load best according to AWS's analyses. There are alternatives —"equal" jitter (half + random(0, half)) or "decorrelated"— but full jitter is the simplest and the one that works best in most cases. The cap stops the fourth retry from waiting eight minutes.

The practical distinction —5xx yes, 4xx no— has two important nuances. The 429 is a 4xx that is retried, because it means exactly "come back later", and it has to be done with more wait than usual (respect the Retry-After header if it comes with the response). And timeouts that expire are the dangerous case: you do not know whether the operation ran or not. Retrying them is correct only if your operation is idempotent; if it is not, a retry after a timeout is exactly how a customer ends up being charged twice.

A very common mistake is treating DynamoDB's ConditionalCheckFailedException as transient. It is not: it means the condition was not met —normally, that it already exists— and retrying will give the same result. In the idempotency pattern, that error is the correct answer, not a failure.

There are three layers of retry and they stack up, which surprises a lot of people.

The danger is the product. If the SDK retries 5 times, the consumer receives the message 4 times (maxReceiveCount=4) and the workflow's Retry tries 4 more, a 30-second outage can generate 80 calls for a single order. With 450 orders in flight that is 36,000 calls against a service that was already in trouble.

MercadoFresco's discipline: one layer is in charge and the rest are minimal. In the order process, the layer in charge is the Step Functions Retry, which is where the business policy lives; the SDK is left at max_attempts=2 to absorb instantaneous network failures, and the queues' maxReceiveCount acts only as a safety net against failures of the consumer itself. Write down somewhere how many total attempts a dependency can receive in the worst case: if the number is above 15, something is wrong.

The retry storm is the most common cascading failure in distributed systems, and it deserves describing in full because it is nearly always diagnosed the wrong way round:

1. The payment gateway degrades: latency rises from 240 ms to 4 s.
2. Consumers wait longer, so the processing rate drops and the queue grows.
3. Lambda sees the queue growing and scales: from 20 to 60 pollers.
4. The gateway gets three times the requests just when it is worst, and starts returning 503s.
5. Every 503 triggers retries. The load multiplies again.
6. The gateway goes down completely. All the retries fail and consume maxReceiveCount.
7. Thousands of legitimate orders end up in the DLQ.
8. The gateway recovers. All the pending retries come out at once and knock it over again.

Four defences, and all four are needed:

• Jitter on every retry. It breaks the synchronisation of steps 5 and 8.
• A concurrency cap (MaximumConcurrency on the event source mapping). It prevents step 3: the queue grows, but the pressure on the gateway does not.
• A retry budget. Retry at most a percentage of the traffic (10 % is usual). If more than 10 % of the calls are retries, retrying stops until the proportion comes down.
• A circuit breaker. Stop calling altogether while the service is down.

The circuit breaker is a component that counts failures and, when they go past a threshold, stops calling the service and fails immediately. It sounds drastic and it is exactly what is needed: if the gateway has had 20 failures in a row, request number 21 is going to fail as well, and the only thing it achieves is to burn 30 seconds of timeout and keep the pressure on a service that is trying to recover.

The state is stored in mercadofresco-catalogo (ElastiCache/Valkey, 06-05), because it has to be shared between all the consumers: a breaker held in process memory is worth nothing when there are 20 concurrent Lambdas, each with its own counter.

What matters is not the code, it is what you do when the circuit is open. Failing fast is only worth something if there is a plan:

• The charge cannot be degraded: if the gateway is down, the order is not confirmed and the customer is told to try again in a few minutes. Better than 30 seconds of waiting and a generic error.
• The notification to the ERP can wait: the message goes back to the queue with ChangeMessageVisibility(300) and is retried in five minutes, without spending receives.
• The confirmation email can be degraded to a secondary provider, or simply delayed.

And the breaker must be observable: every transition to open publishes a metric in MercadoFresco/Tienda and an alarm tells Marta through alertas-mercadofresco. An open breaker that nobody sees is a feature that has been switched off in silence.

A message in a DLQ is not an error: it is an unanswered question. The first thing is to classify it, because the three types demand completely different actions.

The way to tell them apart in 30 seconds: look at the distribution over time. If the 340 messages came in between 18:12 and 18:41, it is transient and a redrive fixes it. If they trickled in over three days, it is poison or data, and a redrive will only send them back to the DLQ.

VisibilityTimeout=5 is the detail that turns this into a safe tool: the messages go back to the DLQ straight away and are not lost if the script dies. Never inspect a DLQ by deleting messages.

And the alarm, which we already set up in 07-01 but which now makes complete sense: ApproximateNumberOfMessagesVisible > 0 on the DLQ is the most valuable health signal in the whole module, because it is the only one that says "there is paid-for work that has not been done". It goes to alertas-mercadofresco with a threshold of 0 and a period of 5 minutes.

A runbook is a written procedure that somebody can follow at 3 in the morning without thinking. This is MercadoFresco's, and it lives in the repository next to the queue definitions.

1. Contain. Is it still growing? Look at the DLQ's ApproximateNumberOfMessagesVisible and the source queue's ApproximateAgeOfOldestMessage. If it is growing, the problem is alive: the redrive can wait; first you stop the bleeding. If it is the dependency that is down, consider temporarily disabling the event source mapping so that messages pile up in the queue —which keeps them for 4 days— instead of exhausting retries and falling into the DLQ.

2. Classify. Run inspect_dlq over 10 messages. Note down: same time window or a trickle? Same error? Uniform ApproximateReceiveCount? Decide poison, transient or data.

3. Diagnose. Search CloudWatch Logs by the MessageId of two or three messages to see the real exception. Cross-reference the time window with the dependencies' metrics and with trail-mercadofresco in case there was a configuration change.

4. Fix. Depending on the type: deploy the fixed consumer, wait for the service to come back, or correct the data at source. There is no redrive without a fix first; returning messages to a queue whose consumer is still broken only duplicates the work and fills the logs.

5. Reprocess. With a rate limit, always:

6. Verify. The DLQ must end up at zero and the source queue's NumberOfMessagesDeleted must go up by the expected amount. If the messages come back to the DLQ, stop and go back to step 2: the fix was not the right one.

7. Record. How many messages, what caused it, what was changed and what would have warned earlier. This step is the one that stops the same incident happening three times.

A warning about reprocessing. Every message that comes back will go through the consumer again, and some of them may have been partially processed before failing. If the consumer is not idempotent, the redrive is a machine for generating duplicates. This is where the first half of this lesson stops being theory.

Ordering is asked for far more often than it is needed, and it costs dearly. Before demanding it, ask three questions: do the messages affect the same data? is the operation relative (add, subtract) or absolute (set)? does the final result change if they are applied the other way round?

Global ordering is extremely expensive and almost never necessary. Demanding it in a FIFO queue with a single MessageGroupId limits you to one message in flight: it does not matter that you have 50 consumers, you process serially. With the sku as the group, MercadoFresco has 3,400 groups, guaranteed ordering where it matters and parallelism of 3,400 where it does not.

Choosing the group key is the same decision as the DynamoDB partition key (06-02): as granular as the ordering allows it to be. For stock movements, the sku. For the life cycle of an order —created, paid, prepared, dispatched—, the pedido_id, because the order matters within one order and not between different orders.

There is an alternative that avoids FIFO altogether and is worth knowing: make ordering irrelevant. If every message carries a timestamp or a version number and the consumer discards anything older than the current state —a conditional write of the kind if version > current_version— out-of-order messages resolve themselves. It is more work in the consumer and vastly better for performance. It is the same idea as idempotency: a robust consumer is worth more than an expensive guarantee from the transport.

In 07-01 we left this hole open. The code was:

There is no transaction spanning Aurora and SQS. If step 2 fails —network, throttling, the process dies— the order exists in the database and nobody hears about it: it is not prepared, no email is sent, it is not delivered. Reversing the order does not help: then the risk is announcing an order that does not exist, which is worse. This is the dual write problem, and it is not solved with retries, because the process can die between the two operations.

Solution 1: an outbox table. The event is written in the same transaction as the order, in a table in the database itself. A separate process reads that table and publishes.

Either the order and its event both exist, or neither of them does: that is what the transaction gives you. Afterwards, a publisher reads the rows with publicado = false, sends them and marks them. It can fail and retry with no problem: it will publish the event twice at most, which is "at least once", which is exactly what the idempotent consumers of this lesson know how to tolerate.

FOR UPDATE SKIP LOCKED allows several publishers in parallel without treading on each other, and ORDER BY creado_en preserves the ordering in case it matters.

Solution 2: change data capture over Streams. If the write store is DynamoDB, the pattern is even cleaner: you write only to the table, and DynamoDB Streams generates the event automatically. There is no dual write because there is only one write. An EventBridge Pipes pipe-mf-... (07-03) reads the stream, filters and publishes to the bus. For Aurora there is an equivalent with Debezium/DMS reading the transaction log, although it is considerably heavier to operate.

MercadoFresco uses both: outbox in Aurora for the order events, and Pipes over the mercadofresco-carritos Streams for CarritoAbandonado. And the outbox table needs its own cleanup —deleting what was published over 7 days ago— or it will grow unchecked, like the zombie baskets of 06-02.

The queue as a buffer. On Friday 900 orders an hour come in, in bursts; the consumers process 600. Without a queue, the 300 of difference would be 503 errors in the customer's face. With a queue, the depth rises to 1,200 messages towards 20:00 and comes back to zero by 22:00. The peak is turned into time, which is a far cheaper resource than capacity.

The rule for sizing is simple: the system does not need to withstand the peak, it needs to withstand the average plus a margin, and enough queue for the area under the peak's curve. If the peak lasts 4 hours with an excess of 300 messages/hour, the queue will reach some 1,200 messages: perfectly normal.

Concurrency limits. Aurora aurora-mf-escritor withstands around 200 connections. If Lambda scales to 400 concurrent invocations, each with its own connection, the database refuses connections and everything fails, including the shop. Three layers of defence:

Controlled throttling. When the one getting saturated is a third party, the rate has to be limited from our side: invocation-rate-limit-per-second on EventBridge API destinations (07-03), throttlePolicy.maxReceivesPerSecond in SNS delivery policies (07-02), MaxConcurrency in a Step Functions Map (07-04). They are all the same idea: it is better to go slowly on purpose than to go fast and cause an outage.

And a hygiene rule: backpressure upwards, never downwards. When the system is saturated, the right answer is to let the queue grow and raise the alarm, not to increase parallelism against a dependency that is already suffering.

Three combinations that solve most real cases: SNS→SQS (fan-out with durability), EventBridge→SQS→Lambda (content-based routing with buffering) and EventBridge→Step Functions (a fact starts a process). And one that is almost never a good idea: SNS→Lambda directly for work that matters, for all the reasons given in 07-02.

What has been gained, measured:

Sixty-two dollars a month —SQS almost free, SNS 2.60, EventBridge 0.95, Step Functions 54, DynamoDB 0.90— in exchange for confirming an order being one fast, reliable thing again, and for the rest of the world finding out at its own pace.

Using the MessageId as the idempotency key. It changes if the producer resends the same work, so it does not protect against the most frequent case. The key derives from the domain: cobro:<pedido_id>.

An idempotency table without a TTL. It grows indefinitely and you pay storage for records nobody will ever query. A 24 h TTL.

Idempotency with only two states. Without EN_CURSO, two simultaneous consumers both see "it is not there" and both run. The conditional write with an expiring EN_CURSO is what closes the race.

Locking with EN_CURSO without expiry. A consumer that dies halfway leaves that operation locked until the TTL expires, and that order is never processed.

Retrying without jitter. The recovery causes the next outage.

Retrying business 4xx errors. It delays the answer and fixes nothing. Exception: the 429, which is retried, and with more wait.

Stacking retries in three layers. SDK × service × workflow can multiply the load on a degraded dependency by 80. One layer is in charge; the rest, minimal.

Redriving without fixing first. The messages come back to the DLQ, and if the consumer is not idempotent, they duplicate effects as well.

Inspecting a DLQ by deleting messages. Use a short VisibilityTimeout and never delete during the diagnosis.

Asking for global ordering "just in case". A single MessageGroupId turns a distributed queue into a serial process.

Writing to the database and publishing without a transaction. The dual-write problem. Outbox or Streams; there is no third option that works.

Tip: make it idempotent before making it fast. An idempotent consumer lets you retry without fear, reprocess a DLQ, replay EventBridge archives and deploy twice by mistake. It is the property that prevents the most problems per line of code.

Tip: write the runbook before the incident. At 3 in the morning nobody designs a procedure.

Tip: measure the worst case of attempts per dependency. If the number goes above 15, review the configuration: some layer is retrying too much.

A customer complains that they were charged 48.20 € twice. The data: cola-mercadofresco-pedidos with a VisibilityTimeout of 120 s; the consumer calls the gateway with a read_timeout of 180 s; the SDK is at max_attempts=5; maxReceiveCount=4; there is no idempotency table; the gateway supports Idempotency-Key but it is not used; the logs show a latency spike in the gateway at 19:14.

Answer: (a) the two independent mechanisms that could have duplicated the charge; (b) which is more likely given the numbers; (c) five fixes ordered by effectiveness; (d) which is the only one that eliminates the problem at the root; (e) what would have changed if a FIFO queue had been used.

mercadofresco-pedidos-fallidos has 340 messages. The source queue's ApproximateAgeOfOldestMessage is 12 s (normal). Inspecting 10 messages: all with ApproximateReceiveCount = 5, SentTimestamp spread between Sunday at 22:04 and 23:51, and all with "version": 2 in the body while the consumer expects "version": 1. There was a deployment that Sunday night.

Apply the runbook: (a) steps 1 and 2 with your conclusion; (b) what type of failure it is and why the distribution over time is misleading; (c) the two possible fixes with their pros and cons; (d) the reprocessing command and why the rate limit matters here; (e) what design safeguard would have prevented it.

The shop does an INSERT in Aurora and then a put_events on bus-mercadofresco. Marta finds that 1 in every 900 orders exists in Aurora but generated no event: it was not prepared, the customer was not told and nobody knew until they complained.

Design the solution: (a) why retrying the put_events is not enough; (b) the schema of the outbox table and the SQL transaction; (c) the publisher, stating how often it runs and how it avoids duplicating and treading on others; (d) what delivery guarantee the whole thing offers and what it demands of the consumers; (e) which two extra things have to be operated that did not exist before.

Solution 1

(a) The two mechanisms. One: the visibility timeout is shorter than the processing time. The queue has 120 s and the consumer can wait 180 s for the gateway. If at 19:14 the gateway took 150 s, the message became visible after 120 s, another consumer picked it up and charged in parallel. Two: the SDK's retries on a non-idempotent call. With max_attempts=5, if the gateway charged and then failed to respond —or took longer than the timeout— botocore retries and charges again, all within the same receive of the message.

(b) Which is more likely. The first one. The coincidence between the latency spike and the 120 < 180 relationship is too exact: any call that went past 120 s produced a guaranteed double delivery. The second mechanism probably acted as well, but it requires the call to fail in a particular way, whereas the first only requires it to be slow.

(c) Five fixes by effectiveness.

1. An idempotency table with the key cobro:<pedido_id> in the consumer. It is the only one that makes the duplicate harmless, wherever it comes from.
2. Use Idempotency-Key at the gateway. Almost zero cost, and the guarantee comes from whoever holds the state of the charge. It should have been done from day one.
3. VisibilityTimeout to 400 s, comfortably above the read_timeout of 180 s.
4. read_timeout to 30 s and the SDK's max_attempts to 2. One hundred and eighty seconds is too long for a gateway; if it takes that long, it is better to fail and retry in a controlled way.
5. An alarm on the gateway's p99 latency and a circuit breaker, to stop calling it when it degrades instead of piling up 150-second calls.

(d) The only one that removes the problem at the root is number 1 (with number 2 as reinforcement). Fixes 3, 4 and 5 reduce the probability, but SQS delivers at least once by design: it can duplicate even when everything is perfectly configured. Only an idempotent consumer turns the duplicate into a non-event.

(e) With a FIFO queue. It would have helped little. FIFO deduplicates the producer's send within a 5-minute window, and here the problem was in the consumer: the message was delivered once and processed twice because visibility expired. FIFO's only contribution would be that, with MessageGroupId = pedido_id, there would not be a second message for the same order in flight —but the same message redelivered is still the same message, and the duplicate effect happens anyway—. It is a good example of FIFO used as a substitute for idempotency, which is an expensive mistake.

Solution 2

(a) Steps 1 and 2. Contain: the source queue is 12 s old, which means the problem is not alive. The current consumer is working normally and there is no bleeding; it can be diagnosed without rushing. Classify: the 340 messages came in within a window of 1 h 47 min on Sunday night, coinciding with a deployment, and they all have ApproximateReceiveCount = 5 and "version": 2. Conclusion: they are poison messages, generated by a producer deployed before its consumer.

(b) Type of failure and why it misleads. They are poison —a format failure, not an availability one— but their distribution over time is that of a transient failure: concentrated in a window. What misleads is that the window coincides with the interval in which the new producer coexisted with the old consumer, not with an outage. What disambiguates is the content: "version": 2 against a consumer that expects 1. Rule: the distribution over time is a clue, the content is the proof.

(c) Two fixes. Deploy the consumer that understands version 2 and then redrive. Pro: not a single order is lost and the system ends up in the state it should be in. Con: you have to deploy urgently and verify that version 2 is processed correctly. Write a compatibility consumer that translates version 2 into version 1. Pro: it does not touch the main consumer. Con: it adds a permanent piece for a temporary problem. The first is clearly better; the second only makes sense if the new consumer is not ready and the orders cannot wait. And the heart of the matter: the producer should not have been deployed before the consumer; with version in the body, the old consumer could have ignored the new fields instead of failing, which is precisely why compatible evolution only ever adds fields (07-03).

(d) Reprocessing.

The limit matters because these 340 orders have been stuck for more than 24 hours: processing them hits the gateway, Aurora and the ERP all at once, on top of the normal Monday morning traffic. At 10 a second they take 34 seconds and nobody notices; all at once, they could cause exactly the retry storm this lesson describes.

(e) The safeguard. Two of them, in fact. The compatible evolution rule: never deploy a producer with a new contract before its consumers, and make changes additive only so that an old consumer can ignore what it does not know. And the DLQ alarm, which would have raised the alert on Sunday at 22:09 instead of Monday at 09:00; with it, somebody could have rolled the deployment back in minutes and no order would have waited 11 hours.

Solution 3

(a) Why retrying is not enough. Retries cover the failure of the call, not the failure of the process. If the instance dies between Aurora's COMMIT and the put_events —a deployment, an ASG scale-in, an OutOfMemory, a hardware failure— there is nobody left to retry and there is no trace that something was left unpublished. The probability is low, but 1 in 900 with 240,000 orders a month is 266 orders lost a month.

(b) Table and transaction.

The partial index matters: the publisher's query only looks at unpublished rows, and a full index would grow with the entire history. The transaction is the one from the corresponding section: INSERT into pedidos and INSERT into outbox within the same BEGIN/COMMIT.

(c) The publisher. It runs every second, as a background process on the worker instances or as a Lambda triggered by EventBridge Scheduler. It reads 100 pending rows with FOR UPDATE SKIP LOCKED, which allows several publishers in parallel without two of them taking the same row. It publishes and marks publicado = true. If it dies between publishing and marking, the row will be republished: an acceptable duplicate. It is worth using the outbox row's id as the deduplication identifier downstream, so that the duplicate can be detected.

(d) Guarantee. At least once, end to end. An event is never lost —because it is in the transaction— and it can be duplicated —because the publisher can die between publishing and marking—. That demands that all the consumers be idempotent, which is precisely what this lesson has built. You cannot have an outbox and non-idempotent consumers: it would be swapping a loss problem for a duplication problem.

(e) Two new things to operate. First, the publisher process: it has to be monitored (a metric of pending rows, an alarm if they go above 500 or if the oldest is more than 60 s old) because if it stops, orders go back to not being published and now the failure is silent and global instead of sporadic. Second, the cleanup of the table: a daily delete of the rows published more than 7 days ago, or it will grow out of control like the zombie baskets of 06-02. And as an indirect cost, the event's latency goes up from milliseconds to the second of the polling, which is irrelevant here because everything behind it is asynchronous.

This module started with a shop that did eight things in a row before responding and it ends with an architecture where confirming an order is two operations —charging and writing— and 400 milliseconds. Along the way the pieces have appeared: cola-mercadofresco-pedidos and its siblings to decouple, mercadofresco-pedido-confirmado so that a fact reaches everybody interested, bus-mercadofresco to route by content and receive what AWS itself publishes, and mercadofresco-procesar-pedido to govern the long process with its compensation path.

But what really holds all of that up is none of the four services: it is what this lesson covers. That the system delivers at least once and that the exactly-once effect is put there by your consumer, with a domain key and a conditional write in mercadofresco-idempotencia. That retries need jitter or the recovery causes the next outage, and that stacking them in three layers multiplies the load on whatever is already suffering. That a shared circuit breaker in ElastiCache is worth more than thirty seconds of waiting against a service that is down. That a DLQ is an unanswered question and needs a runbook somebody can follow at 3 in the morning, starting with contain and classify, and with no redrive before fixing. That global ordering is asked for far more often than it is needed and costs you all of your parallelism. That writing to the database and publishing without a transaction loses one event in every nine hundred, and that the outbox or Streams are the only two real answers. And that backpressure —letting the queue grow instead of increasing the pressure— is what turns a peak into time instead of into an outage.

MercadoFresco's architecture is solid by now. And yet there is a problem of the course that remains exactly as it was on day one: all of this has been deployed by hand. The queues were created with aws sqs create-queue typed into a terminal, the rules with put-rule, the state machine by uploading a JSON. Nobody knows for certain whether the test environment has the same configuration as production. Luis still uploads changes over SSH on a Friday afternoon, with the code on his laptop and his fingers crossed. There are no automated tests to say whether a change in the consumer breaks idempotency. There is no way of going back other than copying the previous file, if anybody saved it. And the incident in exercise 2 —a producer deployed before its consumer— is exactly the kind of mistake that a serious deployment process makes impossible.

In module 8, "Developer tools", starting with 08-01, "AWS CodeCommit", we attack the fourth problem of the course: risky deployments. We will see where the code lives and how changes to it are governed, how it is built and tested automatically with CodeBuild, how it is deployed without cutting the service and rolled back on its own with CodeDeploy, and how it is all chained together into a continuous flow with CodePipeline, until we have built an end-to-end pipeline that takes a change of Luis's from his laptop all the way to production without anybody having to type a command at seven o'clock on a Friday evening.