Aurora Libros already builds and deploys itself, but it still lives on one machine. If that machine goes down, the bookshop disappears from the internet. This lesson sets up your first cluster with Docker Swarm: the orchestration closest to what you already know, because it speaks compose.yaml and docker commands.

1. What an orchestrator solves
2. Swarm architecture: managers, workers and Raft
3. Desired state and the reconciliation loop
4. Setting up the cluster: init, join and ports
5. Managing nodes: promote, demote and drain
6. Services and tasks
7. Replicas versus global mode
8. Scaling and placement constraints
9. Overlay networks: VXLAN between nodes
10. The routing mesh
11. Stacks: the compose.yaml you already have
12. The complete deploy section
13. Swarm's native secrets and configs
14. Aurora Libros on a three-node cluster
15. Swarm in 2026: when it is still the right choice

Warning. A cluster multiplies the networking, storage and access-control decisions you have to make: the ports between nodes, the encryption of internal traffic and the location of persistent data must be agreed with the infrastructure and security officer in your organization before anything goes to production.

An orchestrator brings three things Compose on a single machine cannot give you by definition: scheduling (deciding which node each thing goes on), reconciliation (keeping the desired state whatever happens) and networking between machines (letting containers talk even when they are on different hosts).

The cluster state —which services exist, how many replicas, which secrets— is replicated between the managers with the Raft consensus algorithm. Accepting a change requires the agreement of a majority, the quorum, which is (N/2) + 1.

That is where the odd-number rule comes from, and it is not superstition but arithmetic:

The two-manager row is the surprising one: adding a second manager worsens availability, because you need both alive to reach quorum. And there is an important consequence worth being clear about: losing quorum does not stop the running containers, but it leaves the cluster unable to decide anything; you cannot scale, deploy or reschedule until enough managers come back.

Here is the mental shift from everything that came before. With docker run you give orders; with an orchestrator you declare how you want the world to look and it takes care of the rest.

The leader runs a permanent loop: it compares the desired state (what you declared) with the actual state (what exists), and if they differ it creates or removes tasks until they match. When they match, it waits for the next change and compares again.

docker service create --replicas 3 aurora-api does not mean "start three containers": it means "I want there to always be three". Kill one and another appears. Shut down a whole node and its tasks are reborn on the ones that remain. Nobody ran a repair command: the loop simply noticed a difference between desired and actual and corrected it.

The --advertise-addr is mandatory the moment the machine has more than one interface, and leaving it out is the first classic mistake: Swarm picks any old IP —often the public one, or one belonging to a VPN— and the other nodes cannot reach it.

These three ports cause half the trouble in a new cluster, and always for the same reason: 7946 needs TCP and UDP, and 4789 is UDP. A firewall that only opens TCP produces the most baffling symptom possible: the nodes show up as Ready, the services deploy, but containers on different nodes cannot see each other. Never open 2377 to the internet: anybody who reaches it with a token can join your cluster.

The drain → maintenance → active cycle is the basic operating routine of a cluster: the tasks relocate themselves before you touch anything, and nobody is left without service. Watch out for one detail: when you switch back to active, the tasks do not return to the node on their own; they stay where they are until the next deployment.

Swarm introduces two new concepts on top of what you already know:

• Service: the declaration. "I want three replicas of this image with this configuration."
• Task: each unit of work assigned to a node. A task ends up being a container, and it is immutable: it is not modified or restarted, it is replaced by a new one.

docker service ps is the main diagnostic tool, because it also shows the dead tasks and why they died:

That last line tells a complete story: task 3 died on worker-1 with the code 78 you programmed back in 06-01, that is, invalid configuration, and the orchestrator recreated it on worker-2. Without the startup validation, it would show a generic exit (1) there.

Global mode is exactly what the metric and log collectors from module 5 need: every node needs its own cAdvisor and Promtail, and you want them to appear by themselves on every machine you add to the cluster without having to remember anything.

The difference between a constraint and a preference matters a great deal in practice: an impossible constraint leaves the service Pending forever, with no obvious error, whereas a preference only influences the spread. If a service does not start and docker service ps shows no error, suspect a constraint that no node satisfies.

The bridge network from 03-05 only connects containers on the same host. An overlay network connects containers on different nodes as if they shared a cable.

Under the hood it is a VXLAN tunnel: each Ethernet packet from the container is encapsulated inside a UDP datagram aimed at port 4789 on the destination node, which unwraps it and delivers it. As far as the containers are concerned, the existence of two machines is invisible; they carry on resolving aurora-db over DNS and talking to its IP on the overlay network.

The --opt encrypted deserves a conscious decision. VXLAN traffic travels in the clear by default: if your nodes sit on a private network inside a single data center that may be acceptable, but if they cross the internet or a shared network, the catalog, the queries and any credentials that pass through are readable by anybody with access to the medium. This is exactly the kind of decision you must validate with your security officer. Note as well: the encryption applies to the data plane, and it cannot be turned on or off without recreating the network.

When you publish a port in ingress mode (the default), every node in the cluster listens on it, even those running no replica of that service. The node that receives the connection distributes it via IPVS among the live tasks, wherever they are.

The advantage is enormous for the entry load balancer: point it at any node, or at all of them, and it does not need to know where anything is. There are two drawbacks, and they are worth knowing: there is an extra network hop when the receiving node does not hold the task, and the source IP your application sees is the node's, not the real client's, because there is SNAT in between.

host mode is the way out when you need the client's IP (access logging, per-IP limits, geolocation) or the last drop of latency, and it comes at a price: it forces the external load balancer to know which nodes hold replicas, and you cannot have two replicas of the same service on one node, because they would clash on the port.

Here comes the payoff for all of module 4: docker stack deploy accepts your compose.yaml.

The --with-registry-auth is essential with a private registry like ghcr.io: without it the manager understands the credentials but does not forward them to the workers, and the tasks fail with no basic auth credentials on every node except the one where you ran docker login.

The two ignored keys in the middle change how you design the application. Without build, the flow forces you through the registry, which is the right thing in production. And without depends_on, your API has to cope with starting before the database: that is why you implemented the retry with backoff in 04-04, and why the probes from 06-01 return 503 until the dependencies are up.

One nuance that separates reservations from limits and that did not exist in Compose: in Swarm, reservations affect scheduling. The scheduler adds up the reservations of the tasks already placed on each node and only puts a new one where it fits. Over-reserving leaves nodes empty with services stuck Pending; under-reserving piles tasks up until the node runs out of memory. The measured numbers from 06-01 are exactly what you need here.

Secrets travel to the nodes encrypted by mutual TLS, are stored encrypted in the Raft state and are mounted on a tmpfs that never touches disk; exercise 3 verifies this dimension by dimension. On top of that, a secret is immutable: it cannot be modified. Rotating it means creating a new one under another name, updating the service with --secret-rm and --secret-add and deleting the old one. It is deliberately inconvenient: it forces rotation to be a traceable deployment and not a silent change.

configs work the same way but with no encryption at rest, and they are the natural route for init.sql, nginx.conf or prometheus.yml: configuration files that have to reach every node without you copying them by hand.

Warning about the data. The constraint on aurora-db is a patch, not a solution. Local volumes do not travel: if worker-1 dies, Swarm will recreate the PostgreSQL task on another node and find an empty volume, which means the catalog disappears. A real cluster needs network storage (NFS, iSCSI, Ceph, a cloud provider's block volume) or a managed database outside the cluster. Discuss this with your infrastructure officer before putting real data here: it is not a configuration detail, it is an architectural decision.

Swarm is still included in Docker Engine and maintained, but it is a minority option: the industry standardized on Kubernetes, and that is where the managed providers, the tooling and most of the documentation live.

You will do the detailed comparison in 07-02. What is worth holding on to: almost everything in this lesson —desired state, reconciliation, services and replicas, networking between nodes, secrets mounted as files, rolling updates— is exactly the same mental model you are about to use in Kubernetes, under different names. Swarm has not been a detour: it has been the simple introduction to the same problem.

• Two managers. It makes availability worse than one. Always 1, 3, 5 or 7.
• Forgetting --advertise-addr. With several interfaces, Swarm picks badly and the cluster never forms. Always set it explicitly.
• Only opening TCP in the firewall. 7946 needs UDP too and 4789 is pure UDP. The symptom is Ready nodes with a dead overlay.
• Expecting build: to work. Swarm does not build: publish the image first and reference it by tag or digest. And do not forget --with-registry-auth, or the tasks will fail on every node except the one where you ran docker login.
• Local volumes for data. The task is rescheduled and the volume does not follow it. Network storage or an external database.
• Constraints no node satisfies. The service sits in Pending with no clear message. Check with docker node inspect.
• Tip: use docker service ps --no-trunc. The full ERROR column usually contains the exact cause of the failure.
• Tip: label your nodes from day one (zone, disk, data). Relocating services later is then a matter of one constraint.

Exercise 1. Set up a three-node cluster, deploy aurora-api with three replicas and demonstrate reconciliation: kill a container by hand and then drain a whole node, checking in each case where the tasks reappear.

Exercise 2. Demonstrate the routing mesh: publish aurora-web with a single replica, check that it answers from all three nodes and find out which one is actually serving.

Exercise 3. Compare a Swarm secret with an environment variable: try to extract both values from outside the container with docker inspect and from inside through /proc, and explain where each one physically lives.

Solution 1.

The two tests show the same machinery at two scales. When you killed the container, task .1 was recreated on the same node, because the node was still healthy and was the preferred spot; the 137 is the SIGKILL from your docker kill, and the history line with \_ preserves the failed attempt. When you drained worker-2, its two tasks were relocated onto the other two nodes, and the count went back to 3/3 in about fifteen seconds.

What matters is what you did not do: you ran no repair command. You declared "I want three" once, and the reconciliation loop keeps that statement true in the face of a dead container, a drained node or a machine that gets switched off. It is the same mechanism that governs a Deployment in Kubernetes, and it is why restart: always stops making sense here: the restart policy belongs to the service, not to the container.

Notice as well that task .2 ended up on manager-1. With the node.role==worker constraint from compose.swarm.yaml it could not have: it would have stayed Pending until worker-2 came back. That is the trade-off between protecting the managers and having somewhere to reschedule.

Solution 2.

All three nodes answer 200 even though only worker-2 is running the replica. That is the routing mesh: by publishing in ingress mode, every node opens port 80 and forwards over IPVS through the overlay network to wherever the live tasks are.

And there is the side effect you need to know about: the IP recorded is 10.0.0.2, an address on the ingress network, not the real client's. The DNAT in the nat table rewrites the destination and, along the way, the SNAT replaces the source. If Aurora Libros needed the real IP —to rate-limit per client or for analytics—, you would have to publish in mode=host, accepting that only the nodes with a replica would then answer and that the external load balancer has to know that.

The advantage pays off in most cases: the entry load balancer can point at all three nodes without knowing anything about where each service lives, and adding or removing replicas does not force it to change its configuration.

Solution 3.

The asymmetry is total from the very first check: the variable is read in full from the service specification, which anybody with access to the Docker API can query; of the secret you only see the name and the mount point.

The tmpfs on the second line is the key to the design: the secret never touches disk on any node. When the task stops, the filesystem is unmounted and the value ceases to exist; it is not left in an image layer, or a volume, or a log file, or a backup of the node's filesystem.

The inheritance row is the one that causes the most incidents in practice. An environment variable is visible to every subprocess: if your API invokes pg_dump, a deployment script or any third-party tool, that password goes with them, and all it takes is one library logging its environment when it fails —plenty of them do— for the credential to end up in an external error-tracking service. The file in /run/secrets/ is only read by whoever decides to read it, which is exactly what your config.js from 06-01 does, once, at startup.

Aurora Libros no longer depends on a single machine. You know what an orchestrator brings —scheduling, reconciliation and networking between nodes— and you have built a Swarm cluster with docker swarm init --advertise-addr, different tokens per role and the three ports that need opening, with the detail that costs people whole afternoons: 7946 on TCP and UDP, 4789 on UDP. You understand why managers should be an odd number and why two are worse than one, and you can run the drain → maintenance → active routine that lets you touch a machine without anybody losing the ability to buy books.

You have changed mental model: you no longer give orders, you declare a desired state and a loop keeps it true. You verified it by killing a container and draining a whole node, watching the tasks reappear on their own without you running a single repair. You can tell a service from a task, replicas from global mode —the one Promtail and cAdvisor need—, and you know that an impossible constraint leaves a service silently Pending. You have created overlay networks over VXLAN tunnels, with a conscious decision about --opt encrypted for traffic crossing networks you do not control, and you have seen the routing mesh answer from all three nodes with a single live replica, along with its price: the IP Nginx logs belongs to the ingress network, not to the client.

The compose.yaml from module 4 has become a stack with docker stack deploy, with the table of what Swarm ignores —build, depends_on, restart— and the complete deploy section, where reservations are no longer decorative because they govern scheduling. Native secrets have demonstrated their advantage measurably: invisible in inspect, absent from /proc/<pid>/environ, mounted on a tmpfs that never touches disk and not inherited by subprocesses, while the _FILE pattern from 06-01 keeps working without a single line of code changing. And the real limitation is noted in red: local volumes do not travel, so aurora-db is pinned by a node label and that is a patch which has to be replaced by network storage or a managed database before you put real data here.

In the next lesson, Introduction to Kubernetes, you change tools but not ideas. You will see the control plane with its kube-apiserver, its etcd and its scheduler, the fundamental objects —Pod, Deployment, Service, Ingress, ConfigMap, Secret, PVC— with their minimal annotated YAML, the central role of labels and selectors, the kubectl table translated into the Docker commands you already have down, and you will set up a local cluster with kind to take your first steps.