In the previous two lessons we saw what a distributed system is and which models let us describe it. Now it is time to answer the question any Kilometre Zero architect would ask after the Grape Harvest Week outage: is distributing worth it? The honest answer is "it depends", and this lesson provides the tools to turn that "it depends" into a calculation rather than a hunch.

We are going to quantify the advantages (how much capacity we gain by adding machines, how many minutes of downtime per year each "nine" of availability means, how much parallelism really speeds things up) and put a name to the challenges (complexity, heterogeneity, consistency, partial failures, latency, security, observability and cost). We will finish with a table of situations in which it is better not to distribute, because knowing when not to do it matters as much as knowing how to do it.

1. Advantages of distributing
2. Scalability: vertical versus horizontal
3. Fault tolerance and availability: the "nines"
4. Flexibility, locality and resource sharing
5. Amdahl's law: the limit of scaling
6. Challenges of distributing
7. When NOT to distribute
8. Code example: combined availability at Kilometre Zero
9. Common mistakes and tips
10. Exercises
11. Conclusion

The advantages of a distributed system are the flip side of the reasons they exist (lesson 01-01). We group them into five, and we will quantify the first three:

Scaling means increasing a system's capacity to handle more load. There are two routes:

• Vertical scaling (scale up): replacing the machine with a more powerful one (more CPU, more memory, faster disks).
• Horizontal scaling (scale out): adding more machines and spreading the load across them.

Recall the figures from the incident (lesson 01-01): the single server started to degrade at around 380 requests/s and went down above 700 requests/s. Marketing expects the next campaign, "Artisan Cheese Week", with Montblanc Dairy in the leading role, to reach a peak of 1,200 requests/s.

Let's do the sums for both options:

Vertical option. We need a machine that can withstand 1,200 req/s with a 30% safety margin: 1,560 req/s, that is, about 4 times the current capacity (380 "comfortable" req/s). A server four times as powerful as the current one (32 vCPU, 128 GB) does not cost 4 times as much, but typically 5-6 times as much, and it will still be a single point of failure. What is more, it is not clear that the application will make linear use of the extra cores (section 5).

Horizontal option. We measure that one instance of the Python application, on a modest machine (4 vCPU, 16 GB), comfortably handles 200 req/s. So:

Nine small machines during campaign week and two or three for the rest of the year. This elasticity is impossible with vertical scaling.

But beware: the calculation assumes that the application tier is the bottleneck. In the real incident, PostgreSQL ran out of connections at 610 req/s. If we put 9 application instances in front of the same database, we will simply choke it sooner. Scaling the application horizontally forces us to scale the data as well (caching, read replicas, partitioning), which is the subject of Module 4. The lesson: horizontal scaling is not a property of "the system" but of each component, and the system scales only as far as its least scalable component.

The availability of a system is the fraction of time during which it works correctly. It is expressed as a percentage and, colloquially, by its number of nines. It is essential to translate each level into downtime to understand what it really means:

The calculation is straightforward: a year has 365 × 24 × 60 = 525,600 minutes; 0.1% of that figure is 525.6 minutes (8 h 46 min); 0.01% is 52.56 minutes.

Two important observations:

1. Each extra nine multiplies the cost. Going from 99.9% to 99.99% is not "a bit better": it means that over the whole year you can only afford 52 minutes of downtime, including deployments, operating system upgrades, hardware failures and human error. It demands redundancy at every level, zero-downtime deployments and a 24×7 on-call rota.
2. Availability combines. If Anna's request to buy a cheese goes through four components in series (web, orders, inventory, payments), and each one has 99.9%, the availability of the purchase is not 99.9% but roughly 99.6%: the unavailabilities add up. Conversely, if a component has two independent replicas at 99.9% and it is enough for one of them to work, the combined availability rises to 99.9999%. The code example in section 8 performs these calculations.

With the monolith, Kilometre Zero had an actual availability of 99.3% over the last year (around 61 hours of downtime, between Grape Harvest Week, upgrades and a disk failure). The goal after the restructuring is to reach 99.9% for purchases and 99.95% for catalog browsing.

When each service is an independent deployment unit, teams can work without treading on each other's toes. At Kilometre Zero, the delivery team needs to deploy several times a day to tune the routing algorithm, whereas the payments team, which is subject to audits, prefers tightly controlled fortnightly deployments. In the monolith, the two rhythms are incompatible: every deployment is a deployment of everything. Distributing lets each service have its own pace, its own technology (Python for nearly everything, perhaps another language for real-time position processing) and its own owning team.

The speed of light in optical fibre is about 200,000 km/s. A round trip between Barcelona and a data centre in Frankfurt (about 1,100 km) takes at least 11 ms; with real routing hops, 25-35 ms. Between Barcelona and Virginia (USA), some 90-110 ms. If loading a catalog page requires 10 sequential requests, the difference between serving from nearby and from far away is more than a second. That is why large systems replicate data and services across several regions, and use CDNs for static content (the product photos, in Kilometre Zero's case). Serving each city from a nearby node is an advantage a centralised system cannot offer.

A distributed system lets many users and services share expensive resources: a database, an analytical processing cluster, a photo store. Kilometre Zero's analytics service will need enormous computing capacity for a few hours a day to process sales; for the rest of the day, that capacity can be put to other uses. Sharing is more efficient than dedicating.

It is tempting to think that twice as many machines give you twice the performance. That is almost never the case, and the reason was formulated by Gene Amdahl in 1967: every task has a part that cannot be parallelised, and that part limits the maximum speedup.

If a fraction p of the work is parallelisable (and therefore 1 − p is sequential), the speedup S when using n machines is:

And as n tends to infinity:

Let's apply it to Kilometre Zero. Suppose that, when processing an order, 90% of the work (validating, calculating prices, generating the confirmation, rendering) can be parallelised across instances, but 10% (writing to the single PostgreSQL database, which serialises transactions on the stock) is inherently sequential. Then:

With 8 machines we get less than 5 times the capacity, and we will never exceed 10 times however many machines we add, as long as that 10% remains sequential. This ties in with section 2: the 9 application instances will not deliver 9 × 200 = 1,800 req/s if the database is the sequential component. The only way to break through the ceiling is to reduce the sequential fraction: caching catalog reads (04-05), partitioning the data (04-01), or making asynchronous those operations that do not need to be immediate (02-04). At heart, a good deal of distributed architecture consists of attacking Amdahl's (1 − p).

Every advantage comes with a bill. These are the eight challenges that will appear, one by one, in the rest of the course:

It is worth dwelling on consistency, because it is the most counter-intuitive one. In the monolith, the PostgreSQL transaction guarantees that if Anna buys the last unit, Mark will see zero stock. With orders and inventory separated, and perhaps with inventory replicated across several cities, that guarantee disappears by default and has to be rebuilt with specific techniques (consensus, distributed transactions, sagas) that carry a cost in latency and complexity. Module 3 is devoted entirely to this problem; here it is enough to know that it exists and that there is no free solution.

A good architect knows how to say "no". These are situations in which distributing is, almost certainly, a mistake:

After Grape Harvest Week, Kilometre Zero meets the criteria for distributing: the load no longer fits on one machine, there are several teams, the domains (catalog, orders, delivery...) have natural boundaries and an availability of 99.3% is unacceptable for the business. But it is worth noting that not every component has to be distributed in the same way or at the same time: lesson 01-06 proposes a gradual path.

Let's write a small simulator that calculates the availability of a business flow from the availability of the components it goes through. The two rules are:

• Components in series (the operation needs all of them to work): the combined availability is the product of the availabilities. If A has 0.999 and B has 0.999, A and B together have 0.999 × 0.999 = 0.998.
• Components in parallel (it is enough for one to work, for example replicas): the combined unavailability is the product of the unavailabilities. If A and A' have 0.999 each, the probability of both failing at once is 0.001 × 0.001 = 0.000001, so the availability is 1 − 0.000001 = 0.999999.

The second rule assumes that failures are independent, which in reality is not always true (if the two replicas share a power supply or a network, they fail together). We will comment on this after the code.

The output is as follows:

Let's go through the program and its results:

1. Component stores a name and an availability between 0 and 1; yearly_downtime_min translates it into minutes of downtime per year, which is the measure the business understands.
2. in_series multiplies availabilities. Look at scenario 1: no individual component drops below 99.5%, yet the full purchase ends up at 99.2%, almost 70 hours a year. The more components an operation goes through, the worse it gets. This is the hidden cost of "breaking things down into services": every network boundary added to a synchronous flow takes availability away.
3. in_parallel and replicate apply the redundancy rule. Scenario 2 shows the effect: replicating the four services takes the purchase from 99.2% to 99.95%, and now the dominant component is the one that has not been replicated, PostgreSQL (4 h 23 min of the 4 h 37 min total).
4. Scenario 3 replicates the database too and reaches 99.997%. But be careful: replicating a database is not as simple as replicating a stateless service. The copies have to be kept consistent, and that is the whole of Module 3. The figure is an optimistic upper bound.
5. The last block answers a very practical question: payments, which depends on an external gateway with only 99.5%, goes up to 99.9975% with two replicas. That, however, is only true if the two replicas fail independently. If both depend on the same external gateway and that is what goes down, both go down together and the replica is useless. The model in this script does not capture correlated failures; in reality, you always have to ask what the replicas share (network, power, provider, a software version with the same bug).

• Adding availabilities instead of multiplying them. "Each service has 99.9%, so the system has 99.9%" is false. With five components in series at 99.9%, the system has 99.5%. Every network hop in a synchronous flow is a multiplying factor smaller than one.
• Believing that replicas fail independently. Two replicas in the same rack, with the same software version and the same external dependency, have highly correlated failures. Real redundancy requires diversity (different zones, staggered deployments).
• Scaling the wrong component. Adding application instances when the bottleneck is the database achieves nothing (Amdahl). Measure before you scale.
• Confusing "faster" with "more scalable". A distributed system is usually slower for a single request (network latency) and more capable for many requests (total throughput). If what matters is the latency of an isolated operation, distribution can make it worse.
• Chasing nines the business does not need. Each nine costs roughly an order of magnitude more. Always ask how much each hour of downtime costs the business and compare it with what it costs to avoid it.
• Tip: document the target availability of each business operation (not of each service) and use the series rule to work out what availability that demands of each component. That way you quickly discover which components need redundancy.

The Kilometre Zero team measures that one instance of the catalog service handles 350 queries/s at 60% CPU, and that above 80% CPU latency shoots up. For Artisan Cheese Week, 2,100 queries/s to the catalog are expected.

1. How many queries/s can one instance handle without exceeding 80% CPU (assuming a linear relationship between queries and CPU)?
2. How many instances are needed for the peak, with N+1 redundancy?
3. If half of the queries could be answered from an in-memory cache without reaching the service, how many instances would be needed?

The business decides that the "check the status of my order" operation must have 99.95% availability. The operation goes through, in series: the gateway (99.99%), the orders service (X) and its database (99.98%).

1. How many minutes of downtime per year does the 99.95% target allow?
2. What minimum availability X does orders need for the operation to meet the target?
3. If one instance of orders only achieves 99.9%, are two independent replicas enough?

The analytics service processes the day's sales in a job that takes 60 minutes on one machine. An analysis shows that 6 minutes are spent reading and consolidating the final result into a single file (sequential) and the remaining 54 on per-city calculations (parallelisable).

1. Calculate the speedup with 6 machines and with 18 machines.
2. What is the minimum possible time, however many machines are added?
3. Modify the script from section 8 (or write a new one) with a function amdahl(p, n) and use it to print a table of speedups for n = 1, 2, 4, 8, 16, 32.

Solution 1:

1. If 350 queries/s use 60% of the CPU, 80% corresponds to 350 × (80 / 60) ≈ 467 queries/s per instance.
2. 2,100 / 467 = 4.5 → 5 instances are needed for the peak, 6 with N+1 redundancy.
3. If the cache absorbs half, 1,050 queries/s reach the service: 1,050 / 467 = 2.25 → 3 instances (4 with N+1). Notice that a cache (lesson 04-05) halves the number of machines: reducing the work is usually more cost-effective than adding machines.

Solution 2:

1. 0.05% of 525,600 minutes = 262.8 minutes (about 4 h 23 min) per year.
2. The availability in series is 0.9999 × X × 0.9998 ≥ 0.9995. Solving for X: X ≥ 0.9995 / (0.9999 × 0.9998) = 0.9995 / 0.99970002 ≈ 0.99980 (99.98%). In other words, orders needs almost the same availability as the overall target, because the other two components already "spend" part of the budget.
3. Two independent replicas at 99.9%: 1 − (0.001)² = 0.999999 (99.9999%), well above the 99.98% required. Yes, they are enough, provided that the failure of one does not drag the other down and that the load balancer in front of them does not become the new single point of failure (it would have to be included in the series chain).

Solution 3:

1. The parallelisable fraction is p = 54 / 60 = 0.9. With n = 6: S = 1 / (0.1 + 0.9 / 6) = 1 / 0.25 = 4.0 (the job takes 15 minutes). With n = 18: S = 1 / (0.1 + 0.9 / 18) = 1 / 0.15 = 6.67 (9 minutes). Tripling the machines only improves the time from 15 to 9 minutes.
2. The minimum time is the sequential part: 6 minutes (maximum speedup 1 / 0.1 = 10). To go below that you have to attack the final consolidation, for example by writing partial per-city results in parallel (the pattern MapReduce follows, lesson 05-02).

Output:

The efficiency column (what fraction of each machine is actually put to use) is the one whoever pays the bill should be looking at: with 32 machines, less than a quarter of each one is being used.

Distributing a system brings real, measurable advantages: horizontal scalability (with a capacity calculation that, in the case of Artisan Cheese Week, replaces one enormous machine with nine modest, elastic ones), availability (where we have learnt to translate the "nines" into minutes of downtime and to combine components in series and in parallel), locality (serving each city from nearby), flexibility (each team at its own pace) and resource sharing. But we have also seen that scaling has a ceiling, Amdahl's law, set by the sequential fraction of the work (usually data access), and that every network boundary we add to a synchronous flow multiplies availability by a factor smaller than one.

The challenges (complexity, heterogeneity, consistency, partial failures, latency, security, observability and operational cost) are the table of contents for the rest of the course. And the "when not to distribute" table is a reminder that the right answer to many problems is still a well-built monolith.

Many of these challenges stem from false assumptions that we programmers make about the network without realising it. The next lesson, The Fallacies of Distributed Computing, names them one by one and shows, with code, what happens when we believe them to be true.