In the previous lesson we split PideYa into services, but we left the underlying question hanging: how do they talk to each other, and what happens when the conversation breaks down? A distributed system is one in which the failure of a machine you didn't even know existed can render your program unusable (Leslie Lamport's famous definition). This lesson covers the communication and data patterns between nodes: from the remote proxy we promised back in module 3 to asynchronous messaging, delivery guarantees, the Outbox pattern, and eventual consistency. The GoF thread continues: here Proxy and Observer are stretched across the network until they are barely recognizable — but the intent is the same.

1. Remote proxy and RPC: the call that crosses the network
2. The 8 fallacies of distributed computing
3. Asynchronous messaging: producer-consumer queues
4. Pub-sub and brokers: Observer at network scale
5. Delivery guarantees and idempotency
6. The Outbox pattern
7. Event-driven architecture in PideYa
8. Eventual consistency and the CAP theorem
9. Complementary resilience: timeout, bulkhead, and dead letter queue

In 03-08 we left a door open: the remote proxy, promised for this lesson. The idea behind RPC (Remote Procedure Call) is that calling another process should look like calling a local method:

That createProxy generates a GoF Proxy (using the dynamic proxies we saw in 03-08) that serializes the arguments, sends them over HTTP or gRPC, waits for the response, and deserializes it. gRPC stubs, Spring's Feign clients, or good old RMI are exactly this: the Proxy pattern with the network inside.

The trap is that the transparency is a lie. A local call fails in one way (an exception); a remote one fails in three: before arriving, while executing, or on the way back with the response — and in the third case the effect did happen even though all you see is a timeout. This "partial failure" does not exist inside a single process, and it is the reason behind almost every pattern in this lesson.

Formulated at Sun Microsystems (Deutsch, Gosling, and others), they are the false assumptions everyone makes at first:

Burn 1 and 2 into your memory: they justify everything that follows.

The alternative to synchronous RPC is not waiting: the sender drops a message in a queue and gets on with its life; a consumer processes it when it can.

This is the producer-consumer pattern: each message is processed by exactly one consumer (the two kitchen workers compete for the messages). Advantages over RPC:

• Temporal decoupling: if the kitchen is down for five minutes, orders wait in the queue instead of being lost.
• Load leveling: the 9 p.m. spike gets queued; consumers work through it at their own pace (and you can add more consumers).
• The price: the producer doesn't know when (or whether) its message was processed — the reply, if there is one, arrives as another message.

Important note: this same pattern exists inside a process with threads and a BlockingQueue; that local version is the subject of the next lesson, 06-04.

In 04-08 we left a door open toward pub-sub; here we walk through it. In publish-subscribe, the sender publishes events to a topic and all subscribers receive a copy. It is exactly the intent of Observer — notifying unknown interested parties without coupling to them — with two differences:

• A broker (a dedicated intermediary) sits between subject and observers, and
• the observers live in other processes and may be down when the event is published.

The two brokers you should know, at a conceptual level:

• RabbitMQ: a classic queue broker; it routes messages to queues and deletes them on acknowledgment. Good for distributed work and asynchronous RPC.
• Kafka: a durable, partitioned event log; messages are not deleted when read, each consumer remembers its position (offset) and can replay the past. That pairs it naturally with the Event Sourcing we saw in 06-01.

How many times does a message arrive? The three possible answers:

The practical choice is almost always at-least-once + idempotent consumers: you accept duplicates and make processing twice harmless. It is the same idempotency discipline we applied to the retries in 06-02, now on the consumer side:

The "exactly-once" some platforms advertise is, in practice, at-least-once with automatic deduplication: idempotency doesn't go away, it just moves.

A subtle, lethal problem: the Orders service must (a) save the order in its DB and (b) publish OrderConfirmed to the broker. Those are two different systems: if it saves and then crashes before publishing, the order exists but nobody hears about it (the kitchen never prepares it). There is no transaction spanning DB and broker.

The Transactional Outbox pattern solves it with a single transaction... in the DB:

1. In the same transaction that saves the order, the event is inserted into an outbox table.
2. A separate process (the relay) reads the outbox table, publishes the events to the broker, and marks them as sent.
3. If the relay crashes, it retries: at-least-once delivery, which we already know how to handle with idempotency.

Atomicity guaranteed by the DB, delivery guaranteed by the retry. Tools like Debezium act as the relay by reading the database's transaction log directly (Change Data Capture).

Let's put the pieces together: the order's journey, which in 05-02 was a sequence of calls inside the monolith, is now a chain of events:

Notice three things. First: nobody calls anybody — each service publishes facts and reacts to facts, like the observers of 04-08 but with no subscriber list in memory. Second: adding a statistics service (our old StatsPanel) means subscribing to the topics, without touching a single line of anyone else. Third: this is a choreographed saga of the 06-02 kind, seen from inside the pipes.

In this world, when the customer asks "where is my order?", the screen can run a few seconds behind reality: the OrderReady event exists but hasn't reached the read view yet. That is eventual consistency: if writes stop, all replicas eventually converge — but in the meantime, you can read the past.

The CAP theorem (Brewer) explains why this is not a fixable defect, at an introductory level: under a network Partition (nodes cut off from each other, and fallacy 1 guarantees it will happen), a distributed system must choose between Consistency (rejecting operations to avoid diverging) and Availability (answering with possibly stale data). PideYa chooses case by case: order tracking prefers availability (better a status 5 s stale than an error), the charge prefers consistency (better to reject than to charge twice).

Three patterns that round out 06-02's circuit breaker and retry (we won't repeat those here):

• Timeout: every remote call carries an explicit wait limit. Without a timeout, a hung service hangs you too; 06-02's circuit breaker counts timeouts as failures. Rule: the caller's timeout must be longer than the callee's, or you will cancel work that was about to arrive.
• Bulkhead (named after a ship's watertight compartments): isolate resources per dependency — one connection pool for calling Payments and a separate one for the Catalog. If Payments gets stuck, it exhausts its pool, not the rest of the application's. It is the intent of isolating so failure doesn't spread, a sibling of the confinement we will see in concurrency.
• Dead Letter Queue (DLQ): when a message fails repeatedly (a malformed event, a consumer bug), retrying it forever blocks the queue. After N attempts it is set aside in a "dead letters" queue where a human or a process examines it. Without a DLQ, a single poison message can stop PideYa's entire kitchen.

• Believing RPC's transparency: treating a remote call as local, with no timeout and no plan for partial failure. Every call that crosses the network needs: a timeout, a retry policy (only if idempotent), and defined behavior when it fails.
• Publishing to the broker outside the transaction: the silent bug that motivates the Outbox. If you save to the DB and then publish "by hand" afterwards, sooner or later they will diverge. Symptom: orders that exist but no other service knows about.
• Assuming a global order of events: brokers usually guarantee ordering only per partition/key. Design consumers to tolerate OrderReady arriving before OrderCharged, or partition by order id.
• Ignoring duplicates "because they almost never happen": with at-least-once, duplicates are a matter of time. Deduplication by event id must be there from day one, and in the same transaction as the effect.
• Fat events: publishing the full order with the menu embedded (fallacy 3). Publish facts with ids and the bare minimum; whoever needs more can query for it.
• The DLQ as a black hole: parking messages is fine; not monitoring the DLQ turns visible failures into invisible losses. Alarm when the DLQ grows.

1. Partial-failure diagnosis. Orders calls Payments over RPC with a 3 s timeout. The call throws TimeoutException. List the three possible scenarios of what happened in Payments and what Orders should do to retry safely.
2. Queue or topic? For each PideYa need, choose between a producer-consumer queue and a pub-sub topic, and justify it: (a) distributing kitchen tickets among the three instances of the kitchen service; (b) telling notifications, statistics, and billing that an order was delivered; (c) processing pending refunds one by one.
3. Apply the Outbox. Write the pseudocode (or sketch Java) for the Orders service's confirmOrder method using the Outbox pattern, and for the relay that publishes. State which delivery guarantee results and what the consumer must therefore do.

1. Scenarios: (a) the request never reached Payments — no charge happened; (b) it arrived and Payments failed midway — there may or may not be a charge; (c) Payments charged correctly but the response got lost or arrived late — the charge did happen. Since Orders cannot tell them apart, the retry is only safe if charge is idempotent: it is resent with the same idempotency key (the order id) and Payments returns the already-recorded result if the charge existed.
2. (a) Queue: each kitchen ticket must be processed by exactly one consumer; the instances compete and also level the load. (b) Pub-sub topic: the same fact interests several independent subscribers, and tomorrow another can be added without touching the publisher — Observer at network scale. (c) Queue: work to be distributed with individual processing, plus the option of a DLQ for refunds that fail repeatedly.
3. confirmOrder: open transaction → orderRepository.save(order) → outbox.insert(new OrderConfirmedEvent(eventId, orderId)) → commit (both writes are atomic because it is the same DB). Relay (loop or CDC): read pending rows from outbox → publish to the broker → mark as sent; if it crashes after publishing but before marking, on restart it publishes again. Result: at-least-once — the consumer must deduplicate by eventId (recording processed ids in the same transaction as their effect).

We have toured the plumbing that holds distributed PideYa together: the remote proxy that dresses the network up as a method (and the eight fallacies that punish whoever believes it), the queues that decouple in time, the pub-sub that carries the Observer's intent to planetary scale, the delivery guarantees with their idempotency toll, the Outbox that ties DB and broker together, and the eventual consistency that CAP makes inevitable. All of this is about coordinating processes separated by a network. But there is a closer, more treacherous front left: inside each of those services there are threads sharing memory, competing for the same state nanoseconds apart. That synchronized and that double-checked locking we tiptoed past in the Singleton will finally get a real explanation: Concurrency Patterns.