The previous lesson ended with the most uncomfortable question of the module. In the monolith of 01-06, creating an order was one transaction: BEGIN, insert the order, deduct the stock, record the charge, COMMIT; if any step failed, ROLLBACK and it was as though nothing had happened. Today that same operation crosses three services with three databases, km0_orders, km0_inventory and km0_payments, each replicated as we saw in 03-04, and no PostgreSQL transaction spans all three. If Mark's charge is rejected after the last aged-cheese has been deducted, somebody has to put it back in stock; if orders dies between the deduction and the charge, Mark's order is left in a limbo nobody designed.

This lesson presents the two families of answers. The first tries to preserve the monolith's atomicity with a two-phase commit protocol (2PC), which PostgreSQL supports through PREPARE TRANSACTION; we will see how it works, why it blocks when the coordinator crashes and why microservices avoid it. The second gives up atomicity and replaces it with a saga: a sequence of local transactions, each with its compensation, coordinated by events (choreography) or by an orchestrator with persisted state. We will implement the orders orchestrator in Python, with its sagas table in km0_orders, and watch it compensate the stock when payments rejects Mark; then we will write the same saga as a choreography with Kafka consumers and compare the two. We will finish with what sagas lose compared with ACID (isolation) and the countermeasures, and with the TCC pattern. The conclusion closes Module 3 and opens Module 4, devoted to where and how the data we now know how to replicate and coordinate is physically stored.

1. ACID and the transaction that no longer exists
2. Two-phase commit (2PC)
3. Why 2PC blocks, and 3PC
4. Real-world 2PC: XA and PREPARE TRANSACTION in PostgreSQL
5. Why microservices avoid 2PC
6. Sagas: local transactions with compensations
7. Choreography: the saga as a chain of events
8. Orchestration: order_saga.py with persisted state
9. Choreography vs orchestration table
10. Compensations, idempotency and the lost isolation; TCC
11. Common mistakes and tips
12. Exercises
13. Conclusion

Let us recall what the monolith's transaction promised, because each letter of ACID is going to meet a different fate in the distributed system:

The original transaction, in km0/sql/monolith.sql from 01-06, was essentially this:

If the UPDATE affected no rows (out of stock) or the payment gateway failed, a ROLLBACK undid everything. Today the three statements live in three services, and the question is what replaces the COMMIT.

Two-phase commit (Gray, 1978) is the classic atomic commit protocol: getting several participants, each with its own local transaction, to all commit or all abort. Its roles:

• Coordinator: the process that drives the protocol (in our case it would be orders, or an external transaction manager).
• Participants: the databases or services with an open local transaction (km0_orders, km0_inventory, km0_payments).

Phase 1, prepare. The coordinator asks each participant "can you commit?". A participant that answers yes commits itself irrevocably: it writes the transaction to disk in such a way that it can commit it even after a restart, and it holds its locks until the decision arrives. A participant may answer no (constraint violation, out of stock, card declined), and then the decision will be to abort.

Phase 2, commit or abort. If everyone has said yes, the coordinator writes the decision to its own log (this is the point of no return: from here on the transaction is committed even though nobody knows it yet) and sends commit to everyone; if anyone said no, it sends abort. The participants carry out the decision and reply; the coordinator can retry phase 2 as many times as necessary, because a participant in the "prepared" state can always obey.

The protocol's correctness rests on two promises: the prepared participant never decides on its own, and the coordinator never forgets a decision it has made. Both require disk writes (fsync) at the right moment, which is why 2PC costs at least two network round trips and two fsync calls per participant, on top of the work itself.

The weak point lies in the word "never" in the first promise. Imagine that all three participants have answered yes and the coordinator dies immediately afterwards, before sending the decision (or even before writing it). The participants are in the prepared state: holding their locks (the stock row for aged-cheese locked, Mark's order invisible to reads that need the lock) and unable to decide anything:

• They cannot abort, because the coordinator may have decided to commit and already told another participant.
• They cannot commit, because another participant may have said no.
• Asking the other participants does not always help: if they are all prepared, none of them knows the decision.

All they can do is wait for the coordinator to come back and read its log. If the coordinator takes an hour, that stock row is locked for an hour; this is what is known as the blocking problem of 2PC, and it is an availability problem in the sense of 03-02: the system sacrifices A (live participants that cannot make progress) to preserve atomicity. In practice, operators end up resolving prepared transactions by hand (deciding commit or abort by inspection), which is called a heuristic decision, with the obvious risk of deciding differently from the coordinator.

Three-phase commit (3PC, Skeen, 1981) adds an intermediate phase (pre-commit) so that the participants can work out the decision without the coordinator, and eliminates blocking... at the cost of assuming a synchronous network with bounded delays and no partitions, precisely what 01-02 and 03-02 told us we do not have. With a partition, 3PC can lead one side to commit and the other to abort. That is why hardly anybody uses it, and why the modern solution to 2PC blocking is to make the coordinator fault-tolerant through consensus (03-03): Spanner, for example, runs 2PC between Paxos groups, so that the "coordinator" is a replicated group that does not die. It is correct and it is very expensive.

The XA standard (X/Open, 1990s) defines the interface between a transaction manager and the participating resources (databases, queues); Java exposes it as JTA, and classic application servers used it for transactions that spanned a database and a JMS queue. PostgreSQL implements the participant side with three statements: PREPARE TRANSACTION 'id' (phase 1: the current transaction becomes prepared and survives restarts), COMMIT PREPARED 'id' and ROLLBACK PREPARED 'id' (phase 2). It requires max_prepared_transactions > 0 in postgresql.conf (the default is 0, deliberately). A minimal coordinator in Python between km0_inventory and km0_orders:

To see the blocking with your own eyes, kill the process (or add a sys.exit()) right after phase 1 and run this query on either database:

The prepared transaction survives even a server restart, and for as long as it exists, the aged-cheese row is locked: any other UPDATE on it waits indefinitely. Only an explicit COMMIT PREPARED or ROLLBACK PREPARED releases it, and deciding which is the heuristic resolution of section 3. (The equivalent of PREPARE TRANSACTION exists in MySQL as XA PREPARE, and in queues such as ActiveMQ or IBM MQ; Kafka does not take part in XA, and neither does RabbitMQ, which already greatly limits its use with our architecture.)

With 2PC working in two lines of SQL, the natural question is why not use it for Mark's order. The reasons are all practical:

1. Coupling: the coordinator needs direct transactional access to the databases of inventory and payments, which breaks the rule "each service owns its data" from 01-06. The alternative, having each service expose prepare/commit/abort operations over gRPC, is possible but turns every service into an XA resource manager, with its logs, its recovery and its orphaned transactions.
2. Availability: the blocking of section 3 means that a crash of orders (the coordinator) leaves rows locked in inventory and payments. A failure in one service becomes a failure in three: the opposite of the fault isolation we were after when we split the services (01-03).
3. Latency and throughput: two phases with an fsync on every participant, with locks held throughout the exchange, limit the number of orders per second to whatever the slowest participant can bear.
4. Heterogeneity: payments talks to an external gateway (02-05) that takes part in no 2PC: you cannot "prepare" a card charge. Kafka and Redis are not XA participants either. As soon as one step is not a relational database, 2PC no longer covers the whole operation.
5. Operations: orphaned prepared transactions require manual intervention, and teams that have run XA in production tend to describe it as the most frequent source of night-time incidents.

2PC is still the right tool inside a system that controls everything (a distributed database such as Spanner or CockroachDB uses it between its own nodes, with a coordinator replicated through consensus), but between independent services the industry has converged on the alternative that gives up atomicity: the saga.

The saga pattern (García-Molina and Salem, 1987, for long-lived transactions in a single database) was reformulated for microservices a decade ago. A saga is a sequence of local transactions T1, T2, ..., Tn, each in one service and committed separately, such that:

• If they all succeed, the business operation is complete.
• If Ti fails, the compensating transactions C(i-1), ..., C1 of those already committed are executed, in reverse order, leaving the system in a state that is semantically equivalent to the initial one (not identical: the cancelled order still exists as a record, the refunded charge shows up on Mark's statement).

For the Kilometre Zero order:

Steps fall into three types, and the order in which they are placed matters:

• Compensatable: they have a compensation (T1, T2). They go first.
• Pivot: the step that decides whether the saga will succeed; once it has committed, the saga cannot abort (T3, the charge: if the charge goes through, the order ships). It is usually the step most likely to fail or the one that cannot be cleanly compensated, and it is placed as late as possible.
• Retriable: after the pivot, steps that cannot fail permanently, only transiently, and are retried until they work (T4).

There are two ways of coordinating who executes each step and each compensation, and they are the subject of the next two sections.

In a choreographed saga there is no coordinator: each service reacts to the others' events and publishes its own, over the orders.events topic of 02-04 and 02-05 (or per-service topics). The order's chain of events:

Each service implements its part as an idempotent consumer from 02-05 which, in the same transaction, applies the local effect and writes the next event to its outbox. The inventory fragment, which reacts to order.created and to payment.rejected:

(psycopg.Rollback is the exception that conn.transaction() catches in order to undo the block without propagating it; here we reuse it as an "out of stock" signal. In the same style, payments consumes stock.reserved, charges with the Idempotency-Key from 02-05 and publishes payment.confirmed or payment.rejected; and orders consumes payment.confirmed for T4 and stock.insufficient/stock.released for C1.)

Choreography is attractive for its apparent simplicity: there is no new component, only consumers. Its problems appear as it grows: to find out what state Mark's order is in you have to reconstruct it from events scattered across three topics; adding a step (for example, "assign courier" between the charge and the confirmation) means touching several services; and cyclic dependencies (orders listens to payments, which listens to inventory, which listens to orders) make it hard to reason about the flow as a whole. With three steps it is manageable; with eight, it is not.

In an orchestrated saga, one component, the orchestrator, knows what the sequence is, invokes each step (through synchronous gRPC or by publishing commands), interprets the response and decides on the next step or the compensation. The orchestrator is a state machine whose state is persisted after every transition, so that if the process dies, another instance (or the same one on restart) picks it up where it left off. At Kilometre Zero it lives in orders, because that is the service that owns the concept of "order" and the one that starts the operation, with this table in km0_orders:

The orchestrator defines the steps as (action, compensation) pairs, moves forward persisting as it goes, and compensates backwards if a step fails permanently. So that the example can be run with no infrastructure, the inventory and payments clients and the saga repository have simulated versions; the real ones would be the gRPC stub from 02-03, the gateway client from 02-05 and psycopg on the table above.

Output:

The important points of the code, for anyone reading it for the first time:

• Persist before carrying on. _transition saves the saga on every state change, and current_step is incremented only after the step has succeeded. If the process dies between step.action(saga) and _transition, on restart resume(saga_id) will run the same step again, which is why every action must be idempotent: reserve with the same reservation_id does not deduct twice (02-03), charge with the same idempotency_key does not charge twice (02-05). The saga directly inherits the work of Module 2.
• The identifiers are decided at the start. reservation_id and idempotency_key are derived from the order_id and stored in data on the first transition, so that a retry after a restart uses exactly the same ones.
• Permanent versus transient failure. A PermanentFailure (out of stock, card declined) triggers compensation; a TransientFailure (gRPC deadline, 503 from the gateway) is retried, and if it happens in a step after the pivot it is left for a periodic process that calls resume on the sagas in progress (the sagas_in_progress index exists for that).
• Compensations are idempotent too (release on a non-existent reservation does nothing) and are executed in reverse order, without touching the steps that never got to run (in saga 2, there is nothing to release).
• T1 and the saga in the same transaction. In the real version, orders_db.create and the first save of the saga go in a single km0_orders transaction (and, if the saga is driven by events, the first command goes to the outbox in that same transaction). That way there can be no pending order without a saga and no saga without an order.

In the three simulated sagas, the outcome is what the monolith would have produced with ROLLBACK: Anna has her cheese and her charge; Mark has neither cheese nor wine, he has not been charged anything, and the wine stock is back at 12. The difference is that it happened in separate steps, visible from outside, and that the cancelled orders exist as records.

At Kilometre Zero, the order saga has three steps today but it will grow (courier assignment, notifying the producer, campaign coupons), and the support team needs to know what state each order is in; that is why the choice is orchestration in orders. Choreography is reserved for simple, stable reactions, such as analytics consuming order.confirmed.

A compensation is a new business transaction with effects of its own, not a ROLLBACK: the refund shows up on Mark's statement, the released reservation generates a stock.updated that catalog will display, the cancelled order stays in the history with its reason. Some actions have no possible compensation (an email that has been sent, a courier who has already set off) and must be placed after the pivot. And every compensation must be idempotent and cannot fail permanently: if refund returns a definitive error, the saga is left in a state that demands human intervention, and you have to design for that (a COMPENSATION_FAILED state with an alert, 07-01).

Sagas give up the I of ACID, and that produces specific anomalies between T1 and T4:

These countermeasures (semantic lock, pessimistic view, reordering, commutative updates, and the "version file" for keeping the previous state) were catalogued by Chris Richardson on the basis of García-Molina's original paper, and most of them boil down to one idea: make it visible that the state is provisional instead of pretending there is isolation.

A variant of the saga that recovers some isolation is TCC (Try-Confirm/Cancel): each step is split into try (reserve the resource provisionally, without consuming it), confirm (consume it for good) and cancel (release the reservation). The orchestrator runs all the try operations, and only if they all succeed does it run the confirm operations; otherwise, the cancel operations. It is 2PC at the business level, with no database locks: inventory with ReserveStock and ConfirmReservation/ReleaseReservation is in fact already a TCC resource, and so is the payment gateway with pre-authorisation and capture. The price is that every service has to implement all three operations and manage reservations that expire (a try with neither confirm nor cancel because the orchestrator died), which makes TCC the choice when resources are scarce and contended (the last unit, a place, a seat) and the plain saga the choice for everything else.

• Using 2PC between microservices "because that is what the database does". It couples, it blocks when the coordinator crashes and it does not cover gateways or brokers. Keep 2PC for the inside of a system that controls everything.
• Sagas with non-idempotent steps. After a restart, the orchestrator re-executes the last step. Without a reservation_id, without an idempotency key, the stock is deducted twice and the customer pays twice. Idempotency identifiers are generated when the saga starts and persisted with it.
• Compensations that can fail with no plan. Design the "compensation failed" state, with an alert and a manual procedure, before it happens in production.
• The pivot in the wrong place. If the step that fails most often (the charge) goes first, little gets compensated; if it goes last, everything before it gets compensated. Order them: compensatable steps, then the pivot, then the retriable ones.
• Pretending there is isolation. Showing the stock deducted by a saga in progress as final, or counting pending orders as sales, produces the errors in the table in section 10. Model provisional states explicitly.
• An orchestrator holding other services' business logic. The orchestrator decides the order and the compensations; it does not decide whether there is stock or whether the card is valid. If it starts doing so, it has become a distributed monolith.
• Sagas with no timeout. A saga sitting in STOCK_RESERVED for hours because payments is not responding holds stock that others would like to buy. Define a maximum time per step and per saga, after which it compensates.
• Tip: keep the saga's history. When a customer asks why their order was cancelled, "charge failed: card declined by the issuer" in the sagas row is worth more than any log.
• Tip: test the orchestrator with test doubles that fail at every step, permanently and transiently, and after a "restart" (calling resume with the saved state). It is the kind of test that we will automate with chaos engineering in 07-06.

Using the simulation in section 8, model a crash of the orchestrator right after inventory reserves Lucy's wine (P-2026-000126, 2 units of crianza-wine, valid card) but before the transition to STOCK_RESERVED. Do it with a SimulatedInventory whose reserve raises a SystemExit exception the first time, after having made the reservation. Then create a new OrderSagaOrchestrator with the same InMemoryRepository and the same simulated services, and call resume(saga.id). Which steps are re-executed? Is the stock deducted twice? What would you change in the implementation so that start returned the saga.id even if the process dies halfway through?

Kilometre Zero wants to add "assign courier" (delivery.assign(order_id, city), compensation delivery.unassign(order_id), may fail permanently if there are no couriers in the city) between the charge and the confirmation. Describe what has to change in the orchestration (section 8) and in the choreography (section 7): which files, which new events, which consumers. Is it right to place it after the charge? What would a permanent failure in that step mean for Mark, whose card is valid this time?

For each operation, choose 2PC, choreographed saga, orchestrated saga or TCC, and justify it in two or three sentences:

1. Moving 50 units of pink-tomato between the stock tables of inv-bcn and inv-vlc (both are PostgreSQL databases of the same inventory service, with no external gateways).
2. Recording a delivery: delivery marks the order delivered, orders changes the status, analytics updates the average delivery time, and an email is sent to Anna.
3. Reserving the last 3 bottles of crianza-wine for an order from Lucy and charging them to a card that requires strong customer authentication (the customer may take minutes to confirm it in their banking app).

Solution 1:

The saga was persisted in STARTED with current_step = 0, so resume re-executes reserve_stock. Since SimulatedInventory.reserve is idempotent by reservation_id (res-P-2026-000126 is already in reservations), it does not deduct twice: the stock stays at 10, Lucy is charged and the saga reaches COMPLETED. If you remove the if reservation_id in self.reservations check, you will see the stock at 8: it is exactly the failure that the idempotency of 02-03 prevents. As for start: in the real implementation it returns nothing to anyone if the process dies; what matters is that the saga is in the table, and that a periodic process (SELECT id FROM sagas WHERE state NOT IN ('COMPLETED','CANCELLED') AND updated_at < now() - interval '1 minute') calls resume on orphaned sagas. In addition, start should create the order and the saga in a single transaction and answer the client "order received, pending" before running the steps, which may take a while.

Solution 2:

Orchestration: a single change in order_saga.py: insert a Step("assign_courier", "COURIER_ASSIGNED", action=delivery.assign(...), compensation=delivery.unassign(...)) into the self.steps list between charge and confirm_order, add city to data, and allow the new state in the sagas table. The inventory and payments services do not change. Choreography: delivery needs a new consumer of payment.confirmed that publishes courier.assigned or delivery.impossible; orders stops confirming on receipt of payment.confirmed and does so on courier.assigned instead; payment.rejected stays the same, but delivery.impossible must trigger two compensations: payments must consume it and refund (a new consumer) and then inventory must release on receiving payment.refunded (a new event and a new consumer). Three services touched, two new events, and the compensation chain gets longer.

After the charge? If "assign courier" can fail permanently, placing it after the pivot (the charge) forces a refund, which is the most visible and annoying compensation for the customer (Mark would see a debit and a credit on his statement). It would be better to place it before the charge (reserving a courier is compensatable and cheap to undo), making the charge once again the last compensatable-or-pivot step. General rule: steps that can fail permanently go before the pivot.

Solution 3:

1. 2PC (or, better, a single transaction): both tables belong to the same service, inventory, so there is no undue coupling; if they are in the same logical database nothing distributed is needed, and if they are two PostgreSQL instances, PREPARE TRANSACTION between them, coordinated by inventory, is appropriate: homogeneous participants, no gateways, a short transaction, and the service itself can resolve orphaned prepared transactions on start-up (by querying pg_prepared_xacts). A saga would be over-engineering.
2. Choreographed saga: delivery publishes order.delivered in the same transaction as its status change (outbox), and orders, analytics and the notification service consume it independently and idempotently. No compensation is possible or necessary (a delivery cannot be "un-delivered"), there is no pivot, and no step depends on the result of another: it is the simple, stable reaction for which choreography is a good fit.
3. TCC: the last three bottles are a scarce, contended resource, and the charge may take minutes because of strong customer authentication. inventory does a try (a provisional reservation with an expiry, say 15 minutes), payments does a try (a pre-authorisation pending the customer's confirmation); when the bank confirms, the orchestrator runs confirm on both (consume the reservation, capture the charge); if Lucy does not confirm in time, cancel on both. During the wait, the catalogue shows the bottles as reserved, neither sold nor available: the explicit provisional state of section 10. A plain saga with an immediate charge cannot wait minutes with the stock deducted without it appearing as sold, and 2PC cannot include the gateway.

The monolith's transaction, with its all-or-nothing COMMIT, does not exist once the order crosses orders, inventory and payments, and this lesson has shown the two ways of living without it. Two-phase commit preserves atomicity with a coordinator that collects votes in the prepare phase and broadcasts the decision in the commit phase; PostgreSQL supports it with PREPARE TRANSACTION, and we have seen it lock rows indefinitely when the coordinator dies with the participants prepared. That blocking, together with the coupling, the latency and the impossibility of including gateways and brokers, is the reason microservices avoid 2PC (3PC does not help without a synchronous network; the real remedy is a coordinator replicated through consensus, which only distributed databases can afford). The saga replaces atomicity with a sequence of local transactions with compensations, ordered into compensatable, pivot and retriable steps. We have implemented it as a choreography, with idempotent consumers and the outbox chaining order.created, stock.reserved, payment.rejected and stock.released, and as an orchestration in order_saga.py, with the sagas table of km0_orders persisting every transition and a re-entrant resume that picks the saga up after a restart thanks to the idempotency of reservation_id and of the charge key inherited from Module 2; the simulation compensated Mark's reservation when payments rejected him and left the stock intact. The table in section 9 justifies the choice of orchestration for the Kilometre Zero order, and section 10 has put a name to what the saga loses, isolation, with countermeasures (semantic lock, provisional states, pessimistic view, commutative updates) and with TCC as a variant for contended resources.

This lesson closes Module 3. We can now say precisely what a replicated store guarantees (consistency models), what has to be sacrificed when the network splits and what consistency costs when it does not (CAP and PACELC), how a group of nodes elects a leader and agrees on values (Paxos, Raft, etcd), how data is copied between nodes and with which anomalies (leader-follower, multi-leader, quorums), and how a business operation crosses several services without a global transaction (2PC and sagas). All of this treats data as though it were already somewhere. Module 4 deals with that somewhere: where and how the orders, the product photos and the events we now know how to replicate and coordinate are physically stored, when there are too many of them for a single node. It starts with the question that comes before any replica: how to spread different data across many nodes, with partitioning and consistent hashing.