The Project | About Us | Contribute | Donations | License

HOME

Almost everything we do on the Internet every day (searching for a website, paying by card, watching a series) works thanks to dozens or thousands of computers cooperating without our noticing. This lesson lays down the vocabulary and the fundamental ideas on which the rest of the course is built: what exactly a distributed system is, why people build them, which characteristics set them apart from a "normal" program, and what they are made of. We will also introduce the case study that will stay with us throughout the course, Kilometre Zero, a marketplace for local producers that starts out as a simple application on a single server and, as a victim of its own success, will have to become a distributed platform.

Getting these foundations right matters because distributed systems are not simply "several computers": they call for a different way of thinking, in which partial failure, concurrency and the absence of a single source of truth are the rule rather than the exception.

What is a distributed system?
Why do distributed systems exist?
Key characteristics
Transparency: hiding the distribution
Components of a distributed system
Everyday examples
Introducing Kilometre Zero: the initial monolith
Code example: simulating nodes that fail independently
Common mistakes and tips
Exercises
Conclusion

What is a distributed system?

There are many definitions, but two are especially useful because they capture complementary aspects:

Formal definition (Andrew Tanenbaum): "A distributed system is a collection of independent computers that appears to its users as a single coherent system." This definition stresses the goal: cooperation with the appearance of unity. Someone using a search engine does not know (or care) how many machines took part in answering their query.
Ironic definition (Leslie Lamport): "A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable." This definition stresses the problem: hidden dependencies and partial failure. In a distributed system, your program can stop working for reasons that have nothing to do with it.

Both definitions are true at the same time, and the tension between them is the essence of this discipline: we want many machines to look like one, yet each machine can fail on its own.

A simple distinction is worth pinning down:

System	Where it runs	How it communicates	Typical failure mode
Centralised (one process)	One machine	In-memory function calls	All or nothing: the process either works or crashes
Concurrent on one machine	One machine, several threads or processes	Shared memory, pipes	All or nothing (with race conditions)
Distributed	Several machines	Messages over a network	Partial: some parts fail while others do not

The essential difference is not the number of computers, but the fact that the only way to communicate is by sending messages over a network that may delay or lose them, and that each node can fail while the others keep running.

Why do distributed systems exist?

Nobody builds a distributed system for fun: they are harder to design, test and operate than a program on a single machine. They get built because there are reasons that justify the effort:

Scale. A single machine has a limit on CPU, memory, disk and network connections. When demand exceeds that limit, the only way out is to spread the work across several machines.
Geography. Users are spread around the world, and the speed of light imposes a minimum latency. Serving each user from a nearby data centre means having machines in several places. On top of that, some data sources are inherently distributed (sensors, mobile devices, branch offices).
Availability and fault tolerance. If the service runs on a single machine, that machine is a single point of failure. With several machines, the system can keep working even if one of them goes down.
Cost. Beyond a certain size, many modest machines are far cheaper than one enormous machine (and an enormous machine, in the end, has a limit too). It also makes it possible to pay only for the capacity actually used.
Organisation. Large companies have many teams; for each team to own its part of the system, with its own deployment cycle, those parts have to be separate.

In lesson 01-03 we will analyse the advantages and the costs of distributing in detail. For now it is enough to be clear that distribution is always an answer to a specific problem, never an end in itself.

Key characteristics

Every distributed system, whatever its size, shares three fundamental characteristics. They follow directly from "several machines communicating through messages".

3.1 Concurrency

Nodes run simultaneously and independently. At Kilometre Zero, while Anna is adding a cheese to her basket from Barcelona, Mark is paying for an order from Valencia and a courier is sending their position from Zaragoza. Nobody takes turns: everything happens at once, and shared resources (a product's stock, for example) have to be coordinated explicitly.

3.2 No global clock

Each machine has its own clock, and these clocks are never perfectly synchronised. If node A records an event at 10:00:00.120 and node B records another at 10:00:00.118, we cannot say for certain which happened first: the difference may be smaller than the synchronisation error. This has deep consequences for anything that depends on "who was first". We will study it thoroughly in lesson 01-05.

3.3 Independent failures

In a monolithic program, if something fails, everything fails, and you usually find out straight away (an exception, a dead process). In a distributed system:

A node can crash while the rest keep running (partial failure).
The network can lose, duplicate or delay messages.
A node can be so slow that it is indistinguishable from one that has crashed.
And worst of all: there is not always a way of knowing what happened. If you send a request and get no response, did the request never arrive? Did it arrive and get processed, with the response lost on the way back? Is the node down or merely overloaded?

This uncertainty is the root of most of the difficulties in this course. Designing distributed systems is, to a large extent, a matter of accepting that partial failure is inevitable and deciding how to behave when it happens.

Transparency: hiding the distribution

Recall Tanenbaum's definition: the system must appear to be a single one. Each aspect of distribution that is hidden from the user (or from the programmer) is called a transparency. The most important ones are:

Transparency	What it hides	Example at Kilometre Zero
Access	Differences in data representation and in how a local or a remote resource is accessed	The code that queries the catalog is written the same way whether the catalog lives in the same process or on another machine
Location	Where a resource physically is	The app calls `catalog.search("cheese")` without knowing which server or which city it runs in
Replication	That several copies of the same resource exist	The client sees "the catalog", even though five copies are serving it
Failure	That a component has failed and recovered (or been replaced)	If one copy of the catalog goes down, requests go to another without Anna noticing
Scaling	That the system grows or shrinks in size	Adding servers during a campaign does not change the way the platform is used
Concurrency	That several users share the same resource	Anna and Mark query the same stock without getting in each other's way
Migration	That a resource moves elsewhere while in use	A session moves from one server to another without dropping the connection

It is important to understand that full transparency is an ideal, not a reality. Hiding the distribution completely has costs (latency, complexity) and in many cases is counterproductive: a programmer who does not know that a call travels over the network will write code that assumes it never fails and always responds quickly. We will look at exactly this problem in lesson 01-04, which is devoted to the fallacies of distributed computing.

Components of a distributed system

Any distributed system can be described in terms of four kinds of building block:

flowchart LR
    subgraph Node_A["Node A"]
        A1[Application]
        A2[Middleware]
        A3[Operating system]
    end
    subgraph Node_B["Node B"]
        B1[Application]
        B2[Middleware]
        B3[Operating system]
    end
    A2 <-- "Protocols (messages)" --> B2
    A3 <-- "Network" --> B3

Nodes. Each of the machines (physical or virtual, containers included) that run a part of the system. A node has its own CPU, memory, disk and clock. There is no shared memory between nodes: only messages.
Network. The medium the messages travel over: cables, switches, routers, wireless links, the Internet. The network is one more component of the system, with its own failures and limitations, although it is often forgotten.
Protocols. The rules that define how messages are structured and exchanged: format, order, what to do when an error occurs. From transport protocols (TCP, UDP) to application protocols (HTTP, gRPC, MQTT), they are covered in Module 2.
Middleware. The software layer that sits between applications and the operating system/network and provides the transparencies: service discovery, remote calls, message queues, replication and so on. It is what spares application programmers from working directly with sockets and bytes. A good part of the course (RPC, queues, distributed databases, orchestrators) is precisely about middleware.

Everyday examples

To be convinced that distributed systems are everywhere, just look at four things we do every day:

System	What it distributes	Transparencies it offers	What would happen in a partial failure
DNS	A hierarchical database of names, replicated across thousands of servers worldwide	Location, replication, failure	Your resolver asks another server; you hardly ever notice
The Web	Content served by millions of servers, with caches (CDNs) close to each user	Location, replication, scaling	One particular website does not load, but the rest of the Internet keeps working
Online banking	Accounts, transactions and cash machines spread over many systems that must stay consistent	Access, concurrency, replication	You can check your balance but not make transfers (partial degradation)
Video streaming	Copies of each film on geographically distributed servers, plus the recommendation and billing systems	Location, replication, scaling	Recommendations go down, but you can carry on watching your series

Look at the last column: in every case, the system is designed to degrade partially instead of going down altogether. That is one of the promises of distributed systems... and also one of the hardest things to achieve.

Introducing Kilometre Zero: the initial monolith

Throughout the course we will work with Kilometre Zero, a fictional online marketplace that connects local producers with consumers in several cities. Its producers include La Vega Farm (seasonal vegetables), Montblanc Dairy (artisan cheeses) and Roble Alto Winery (wines). Its customers include Anna, Mark and Lucy.

7.1 The starting architecture

Kilometre Zero was born like any sensible project: as a monolith. A single Python web application, with all its modules in the same process, talking to a single PostgreSQL database, all on a single server.

flowchart TB
    U1[Anna - browser] --> S
    U2[Mark - mobile app] --> S
    U3[Lucy - browser] --> S
    subgraph S["Single server (8 vCPU, 32 GB RAM)"]
        direction TB
        subgraph APP["Python application (one process)"]
            C[catalog]
            P[orders]
            I[inventory]
            PA[payments]
            R[delivery]
            AN[analytics]
        end
        DB[(PostgreSQL)]
        APP --> DB
    end

This architecture has enormous virtues, and it is worth saying so plainly to avoid the prejudice that "monolith" is a synonym for "bad design":

Simplicity. A call from the orders module to the inventory module is a plain Python function call: no network, no serialization, no timeouts.
Simple transactions. Deducting stock, creating the order and recording the payment all happen in a single PostgreSQL transaction: either everything is done or nothing is.
Trivial deployment. A single artefact installed on a single server.
Straightforward debugging. A single log, a single stack trace.

On a machine with 8 vCPU and 32 GB of RAM, Kilometre Zero comfortably serves its first two cities: around 300 orders a day and peaks of 40 requests per second.

7.2 The first symptom: "Grape Harvest Week"

In September, Kilometre Zero launches its first big seasonal campaign: Grape Harvest Week, with offers from Roble Alto Winery and other wineries, promoted on social media in five cities. The campaign opens on Monday at 10:00, and this is what happens:

Time	Requests/s	Symptom
09:55	45	Normal
10:02	380	Catalog pages take 3-4 s to load
10:06	610	PostgreSQL hits its connection limit; payments start failing
10:09	700+	The Python process runs out of memory and the server stops responding
10:10	0	Total outage. No catalog, no orders, no courier tracking

Notice a telling detail: the traffic spike is caused by people browsing the wine catalog, yet what goes down is everything, including courier tracking, which has nothing to do with the campaign. In a monolith, all the modules share the same fate: the same CPU, the same memory, the same database connection pool.

This incident will be the starting point for the evolution of Kilometre Zero that we will follow throughout the course. It is worth resisting the temptation to jump straight to the solution: first (lessons 01-02 to 01-05) we need to understand what distributing means and which new problems appear; in lesson 01-06 we will draw the target architecture.

Code example: simulating nodes that fail independently

Let's build a small simulation in Python to get a feel for the difference between a total failure and a partial one. We do not need real networks: each node will be represented by a class, and the "network" by method calls. It is a huge simplification, but enough to illustrate the idea.

The scenario: Kilometre Zero has three nodes handling orders (orders-1, orders-2 and orders-3) behind a very simple load balancer. One of the nodes is going to fail in the middle of the campaign.

import random
from dataclasses import dataclass, field


@dataclass
class Order:
    """An order placed by a Kilometre Zero customer."""
    id: int
    customer: str
    product: str
    producer: str


class NodeDownError(Exception):
    """Raised when we try to talk to a node that has failed."""


@dataclass
class OrdersNode:
    """Simulates a machine running the 'orders' service.

    Each node has its own state (the orders it has handled) and its own
    'health'. It shares no memory with the other nodes: if we want to know
    anything about it, we have to 'send it a message' (call a method).
    """
    name: str
    active: bool = True
    handled: list = field(default_factory=list)

    def process(self, order: Order) -> str:
        if not self.active:
            # In real life there would be no exception: the response would
            # simply never arrive. We model it as an exception to keep it simple.
            raise NodeDownError(f"{self.name} is not responding")
        self.handled.append(order.id)
        return f"{self.name} confirmed order {order.id} from {order.customer}"

    def fail(self) -> None:
        """Simulates the machine suddenly powering off."""
        self.active = False


class LoadBalancer:
    """Sends each order to a different node, taking turns (round-robin)."""

    def __init__(self, nodes: list[OrdersNode]):
        self.nodes = nodes
        self._next = 0

    def send(self, order: Order) -> str:
        node = self.nodes[self._next % len(self.nodes)]
        self._next += 1
        return node.process(order)


def generate_orders(count: int) -> list[Order]:
    customers = ["Anna", "Mark", "Lucy"]
    products = [
        ("Seasonal tomatoes", "La Vega Farm"),
        ("Aged cheese", "Montblanc Dairy"),
        ("Crianza red wine", "Roble Alto Winery"),
    ]
    orders = []
    for i in range(1, count + 1):
        product, producer = random.choice(products)
        orders.append(Order(i, random.choice(customers), product, producer))
    return orders


if __name__ == "__main__":
    random.seed(7)  # so that the run is reproducible

    nodes = [OrdersNode("orders-1"), OrdersNode("orders-2"), OrdersNode("orders-3")]
    balancer = LoadBalancer(nodes)

    succeeded, failed = 0, 0
    for order in generate_orders(12):
        if order.id == 5:
            print(">>> Node orders-2 has gone down! <<<")
            nodes[1].fail()
        try:
            print(balancer.send(order))
            succeeded += 1
        except NodeDownError as e:
            print(f"ERROR processing order {order.id}: {e}")
            failed += 1

    print(f"\nResult: {succeeded} orders succeeded, {failed} failed")
    for node in nodes:
        status = "active" if node.active else "DOWN"
        print(f"  {node.name} ({status}): handled {node.handled}")

Let's go through the program step by step:

Order is a plain data container: identifier, customer, product and producer. We use @dataclass so that we do not have to write the constructor by hand.
OrdersNode represents a machine. The key point is that it has its own state (handled) and its own health (active). When active is False, any attempt to use it raises NodeDownError. This models independent failure: a node can go down without the others finding out.
LoadBalancer spreads the orders across the nodes in turn. In a real system this piece would be a proper load balancer; here it is a simple counter.
In the main loop, when we reach order 5 we switch off orders-2. From then on, one in every three orders fails, but the other two thirds carry on being processed as normal.

The output (abridged) looks something like this:

orders-1 confirmed order 1 from Anna
orders-2 confirmed order 2 from Lucy
orders-3 confirmed order 3 from Anna
orders-1 confirmed order 4 from Anna
>>> Node orders-2 has gone down! <<<
ERROR processing order 5: orders-2 is not responding
orders-3 confirmed order 6 from Lucy
orders-1 confirmed order 7 from Anna
ERROR processing order 8: orders-2 is not responding
...
Result: 9 orders succeeded, 3 failed
  orders-1 (active): handled [1, 4, 7, 10]
  orders-2 (DOWN): handled [2]
  orders-3 (active): handled [3, 6, 9, 12]

Compare this with the monolith during Grape Harvest Week: there, a single failure left 0 orders succeeded. Here, the system keeps serving 67% of the orders. That is a partial failure: worse than "everything works", far better than "nothing works".

But notice, too, what the simulation does not solve, which anticipates topics from later lessons:

The balancer keeps sending orders to orders-2 even though it is down. It would need to detect the failure and stop using the node (lesson 07-03).
Orders 5, 8 and 11 have been lost. Should we retry them on another node? And what if orders-2 did manage to process the order before going down? (lessons 02-05 and 07-04).
The state held by orders-2 (order 2) is stuck on a dead node. How do we get it back? (Module 3, replication).
In the simulation we know for certain that the node has gone down because it raises an exception. In reality, all we would see is that it is not responding, and we would have to decide how long to wait (lesson 01-04).

Common Mistakes and Tips

Confusing "distributed" with "lots of servers". What defines a distributed system is that its parts communicate only through messages and can fail separately. Two processes on the same machine talking over sockets already show many of the problems of a distributed system (though not all of them).
Thinking that full transparency is desirable. Completely hiding the fact that a call is remote leads to code that ignores latency and failures. Good designs make distribution convenient, not invisible.
Distributing too early. Kilometre Zero was right to start as a monolith: it allowed the company to launch quickly and cheaply. Distribution comes when a specific problem (scale, availability, organisation) justifies it. Distributing "just in case" adds complexity with no benefit.
Forgetting that the network is a component. When drawing architectures, it is common to draw boxes (the nodes) and arrows (the communication) and to treat the arrows as if they were free and infallible. Arrows fail, take time and cost money.
Assuming that a node that does not respond is down. It may be overloaded, it may be cut off by a network problem, or it may have processed the request with only the response getting lost. This ambiguity will come up again and again in the course.
Tip: whenever you design an interaction between two components, always ask yourself three questions: what happens if the other side does not respond?, what happens if it responds very late? and what happens if my message arrived but the response was lost? If you cannot answer all three, the design is not finished.

Exercises

Exercise 1: Classifying systems

For each of the following cases, say whether it is a distributed system according to the definition in this lesson, and explain in one or two sentences which of the key characteristics (concurrency, no global clock, independent failures) applies and which does not:

A spreadsheet program that uses several threads to recalculate formulas in parallel.
A Kilometre Zero mobile app that stores the basket on the phone and syncs it with the server when there is coverage.
A backup script that copies files from one disk to another on the same computer.
Three couriers sending their position every 5 seconds to the delivery service.

Exercise 2: Transparencies in Grape Harvest Week

Imagine that, after the incident, Kilometre Zero adds two more servers for the catalog and puts a load balancer in front of them. State which transparencies (from the table in section 4) are being provided to the customers (Anna, Mark, Lucy) and which are not provided to the developers of the orders module, which now has to call the catalog over the network instead of through a function call.

Exercise 3: Improving the simulation

Modify the simulation from section 8 so that the LoadBalancer skips nodes that are down: when a node raises NodeDownError, it must retry the same order on the next active node. If no node is active, it must raise a NoNodesAvailableError exception. Check that, with orders-2 down, all 12 orders are processed successfully.

Solutions

Solution 1:

Not distributed. There is concurrency (several threads), but they all share memory and the same clock, and if the process fails, all the threads fail at once. It is a centralised concurrent system.
Distributed. The phone and the server are independent nodes that communicate only through messages, with different clocks, and the phone can lose coverage (a network failure) while the server keeps running. Note that consistency problems will arise (which basket is the "right" one if it was modified in both places?), which we will deal with in Module 3.
Not distributed. A single process on a single machine. If the destination disk fails, that is a local hardware failure, but there are no independent nodes communicating through messages.
Distributed. Each courier is a node with its own clock (the phone's), sends messages over an unreliable network (4G) and can disconnect independently. The delivery service must assume that positions may arrive late, out of order or not at all.

Solution 2:

The customers are given replication transparency (they see "the catalog", even though there are three copies), location transparency (they do not know which server the balancer sends them to), failure transparency (if a catalog server goes down, the balancer sends them to another one) and scaling transparency (adding the third server changed nothing for them).

The developers of orders are not fully given access transparency: they used to call a local function and now they have to make a network request, handle timeouts and connection errors, and serialize data. Nor do they have complete failure transparency: if the balancer or all three catalog servers go down, orders gets an error and has to decide what to do. That is precisely the lesson: distribution can be hidden from the end user, but it can rarely be hidden completely from the programmer.

Solution 3:

class NoNodesAvailableError(Exception):
    """No node in the cluster is active."""


class ResilientLoadBalancer(LoadBalancer):
    """Same as LoadBalancer, but skips nodes that are down."""

    def send(self, order: Order) -> str:
        attempts = 0
        while attempts < len(self.nodes):
            node = self.nodes[self._next % len(self.nodes)]
            self._next += 1
            attempts += 1
            try:
                return node.process(order)
            except NodeDownError:
                # This node is not responding: try the next one.
                continue
        raise NoNodesAvailableError(
            f"No active nodes for order {order.id}"
        )

All you need to do is replace LoadBalancer(nodes) with ResilientLoadBalancer(nodes) in the main program. With orders-2 down, the final output will show 12 orders succeeded, 0 failed, with the orders spread between orders-1 and orders-3.

A further thought: this solution assumes that if a node raises NodeDownError it is safe to retry on another one. In the simulation that is true, because the dead node has done nothing. In a real system, the request might have arrived and been processed before the response was lost; retrying blindly could create the order twice. The solution to this (idempotency) is covered in lesson 02-05.

Conclusion

In this lesson we have laid the foundations of the course. A distributed system is a set of independent nodes that cooperate by exchanging messages over a network and present themselves to the user as a single system. They are built out of necessity (scale, geography, availability, cost, organisation) rather than for fun, because they bring three characteristics that completely change the way we have to reason: real concurrency, no global clock, and partial, independent failures. To make distribution bearable, middleware offers various transparencies (access, location, replication, failure, scaling), which are always partial and come at a cost.

We have met Kilometre Zero in its initial state (a Python monolith with PostgreSQL on a single server) and we have seen how Grape Harvest Week brings it down completely, including modules that had nothing to do with the campaign. The node simulation has shown the bright side (the system keeps serving orders when a node goes down) and has also exposed the problems we do not yet know how to solve.

In the next lesson, Distributed System Models, we will look at the different ways of organising nodes (client-server, multi-tier, peer-to-peer, services), the assumptions we can make about their communication and their failures, and we will start sketching how Kilometre Zero could be reorganised.

Basic Concepts of Distributed Systems

Contents

What is a distributed system?

Why do distributed systems exist?

Key characteristics

3.1 Concurrency

3.2 No global clock

3.3 Independent failures

Transparency: hiding the distribution

Components of a distributed system

Everyday examples

Introducing Kilometre Zero: the initial monolith

7.1 The starting architecture

7.2 The first symptom: "Grape Harvest Week"

Code example: simulating nodes that fail independently

Common Mistakes and Tips

Exercises

Exercise 1: Classifying systems

Exercise 2: Transparencies in Grape Harvest Week

Exercise 3: Improving the simulation

Solutions

Conclusion

Distributed Architectures Course

Module 1: Introduction to Distributed Systems

Module 2: Communication in Distributed Systems

Module 3: Consistency and Replication

Module 4: Distributed Storage

Module 5: Distributed Computing

Module 6: Security in Distributed Systems

Module 7: Monitoring and Maintenance

Module 8: Case Studies and Applications