Up to now, throughout the whole course, there has been one cluster. The rutas-norte-dev, rutas-norte-pre and rutas-norte-pro namespaces lived side by side in it, separated by quotas, network policies and RBAC. It is a perfectly defensible architecture and it is where almost everybody starts.

But there comes a point when it stops being so. It might be a control-plane upgrade that affects all three environments at once; it might be the auditor asking why the cluster running rutas-norte-dev has access to the network where production personal data lives; it might be that the company opens operations in another region and the latency from Portugal to eu-west-1 starts showing up in conversion. At Rutas Norte it was the first of those: an EKS upgrade that went wrong in pre also left pro unreachable for nineteen minutes.

This lesson is about what changes when the platform no longer fits into one cluster: why you end up there, what topologies exist, how you work day to day without applying to the wrong cluster, how Argo CD governs a fleet, how clusters are connected to one another, and the problem with no easy solution when you have to recover from a disaster, which is the data.

Compliance warning. Distributing clusters across regions directly affects the residency of the personal data of Rutas Norte's customers. Replicating bookings-postgres to another region, storing backups outside the main region, or allowing a cluster in one region to query data held in another, are decisions with legal implications that must be reviewed and approved by the compliance officer before being implemented. The examples in this lesson are illustrative.

1. Why a company ends up with several clusters
2. Common topologies and which one to choose
3. Day-to-day work with several contexts
4. Multi-cluster deployment with Argo CD
5. Common configuration and policies: the cluster as a product
6. Connectivity between clusters
7. Disaster recovery
8. The Rutas Norte plan
9. The cost, the complexity and when not to do it

It is not a decision taken one morning. You get there through an accumulation of reasons, and it is worth recognising them so you know which one is pushing you.

Namespaces separate workloads, but they do not separate the control plane. They share the API server, etcd, the scheduler, the CNI, the Ingress controller and the add-ons. Everything that is "cluster-wide" is shared: CRDs, ClusterRole, admission webhooks, operator versions.

In practice, this means a developer deploying a malformed CRD in dev, or an operator consuming memory without limit, can degrade production. At Rutas Norte it was a badly configured validating webhook in dev that, by failing to respond, blocked pod creation across the whole cluster for eight minutes, production included.

With one cluster, the question "what happens if the cluster fails?" has a single, very bad answer: everything goes down. The managed EKS control plane is very reliable, but it is not the only point of failure: an overly broad NetworkPolicy, VPC IP exhaustion, a faulty CNI version or a human error with kubectl delete affect everything inside it.

Rutas Norte operates mainly in the north of the Iberian peninsula, but it is looking at expanding into southern France. A request from Toulouse to eu-west-1 (Ireland) adds around 35-45 ms of round trip on the network alone. Over a ticket purchase with six chained calls, that is nearly 300 ms of added latency that the user notices.

This is the reason that admits no technical debate: if the applicable regulation, a contract with a corporate customer or a decision by the compliance officer requires that certain customers' personal data must not leave a territory, you need infrastructure in that territory. There is no namespace configuration that solves that.

Kubernetes publishes a minor release roughly every four months, and support for each one lasts around fourteen. Upgrading is compulsory, and testing the upgrade on the same cluster that serves production is impossible by definition. With several clusters you can take dev to the new version, let it break whatever it has to break, and upgrade production weeks later with the lesson learned.

Related, but distinct: even upgrades that go well have risk windows. Nodes are replaced, add-ons restart, the CNI is upgraded. With a single cluster, every upgrade is a production event. With two, it is a production event once every two upgrades, because the first has been rehearsed.

Drift deserves emphasis: the problem with having several clusters is not having them, it is keeping them identical. A patch applied by hand to one and not the other creates a difference nobody remembers until it causes an incident months later. Everything in section 5 exists to fight that.

Start with one. A single cluster with well-separated namespaces, quotas, network policies and RBAC solves 80 % of cases and costs a fraction.

The first split should be production against everything else. It gives the most per unit of complexity: it isolates the control plane of the critical thing, allows upgrades to be rehearsed and satisfies any reasonable audit.

The second, if it comes, is imposed by geography or regulation, not by architectural preference.

The management cluster, from the third onwards. With two clusters, Argo CD can live in the production one managing both. With four, you need a neutral place.

This is where accidents happen. A kubectl delete deploy run in the belief that you are in dev when you are in pro is a real and frequent incident.

They are two small utilities that save a lot of time. Installing them is the first thing anyone working with more than one cluster does.

Five layers, from the softest to the hardest. The first three are warnings; the last two are real barriers.

Layer 1: the prompt shows the context. This is essential. Without it, everything else is optional.

A red background when you are in production. It looks trivial and it is the measure that prevents the most accidents.

Layer 2: a context per session, not a global one. The problem with use-context is that it changes the state of every open terminal. A better solution is to isolate the kubeconfig per terminal:

That way, the production terminal is the production terminal, full stop: there is no way for a use-context in another window to change it.

Layer 3: explicit confirmation for destructive verbs.

Layer 4: read-only contexts by default. This is the first real barrier, because it does not depend on anyone's discipline. The joan-pro user in the kubeconfig assumes the rutasnorte-pro-readonly role, mapped to a read-only ClusterRole:

Writing to production requires explicitly assuming another role, with a temporary session and a recorded reason. It is a deliberate thirty-second friction that turns every write into a conscious act.

Layer 5: never write to production by hand. The ultimate goal. With GitOps (10-05), the only thing that applies changes in pro is Argo CD. People read, diagnose and propose changes in Git. Manual writing is reserved for incidents, with the role-assumption procedure and its audit log.

Argo CD manages remote clusters from a single installation. Each cluster is registered as a destination and the Application resources point at whichever one applies.

The labels are the important part: they are what lets you write "deploy this on every production cluster" without listing them.

argocd cluster add creates a ServiceAccount in the target cluster with the necessary permissions and stores its credentials as a Secret in the Argo CD cluster. That Secret is a high-value credential: whoever reads it can deploy to production. It must be protected by strict RBAC and, preferably, encrypted at rest with an externally managed key.

An ApplicationSet generates Application resources automatically from a template and a generator. The cluster generator creates one for each cluster matching a selector.

This deploys the common add-ons (Kyverno, cert-manager, ingress-nginx, node-exporter) on every labelled cluster, with the per-environment differences resolved by Kustomize.

For the business applications, the matrix generator combines clusters with applications:

With two production clusters and six components, this generates twelve Application resources without writing twelve files. Adding a region means registering a cluster with the right label; adding a component means creating a directory.

The repository structure supports two dimensions: environment and cluster.

An important rule: what is the same across clusters lives in base; what differs, and only that, in the overlay. Every line in an overlay is a difference that somebody will have to understand six months from now. If an overlay grows a lot, it almost always means the base is badly factored.

With several clusters, the question stops being "how do I deploy my application?" and becomes "how do I guarantee that the N clusters follow the same rules?".

The idea of the cluster as a product is that the platform team does not deliver "access to a cluster", but a product with a contract: a freshly created cluster already comes with observability, security policies, RBAC, quotas, an Ingress controller, certificate management and backups configured. Nobody has to set anything up, and therefore nobody sets it up differently.

Even with everything deployed by GitOps, drift appears: add-on versions installed by Terraform, emergency patches, differences in the Kubernetes version itself. A periodic report is worth having.

And the check that really matters, done from Argo CD:

Two clusters are two different pod networks. A pod in eu-west-3 cannot resolve bookings-api.rutas-norte-pro.svc.cluster.local in the eu-west-1 cluster, nor reach its IPs. There are three levels of solution, with very different complexity.

The simplest, and the one that solves most cases. User traffic is directed to the appropriate region by DNS, and each region is self-sufficient.

• Latency policy: each user goes to the region that is fastest for them.
• Failover policy: if a region's health check fails, DNS stops resolving to it.
• An important limit: DNS has a TTL. With a 60-second TTL, failover takes between one and several minutes, because some resolvers ignore short TTLs. There is no instantaneous DNS failover.

When a service in one cluster needs to call another in a different cluster (for example, the PostgreSQL replica following the primary), you need real network connectivity: VPC peering or a transit gateway, non-overlapping CIDR ranges, and explicit firewall rules.

The range requirement deserves attention: if both clusters use 10.244.0.0/16 for pods, routing between them is impossible. You have to plan the addressing before creating the second cluster, and it is a mistake you pay for with a rebuild.

Istio, Linkerd or Cilium Cluster Mesh allow a Service in one cluster to be reachable transparently from another, with mTLS, load balancing across regions, locality failover and unified observability.

What they really solve:

• Transparent discovery: bookings-api.rutas-norte-pro.svc works from any cluster in the mesh.
• Locality failover: the local instance is preferred and it overflows to the remote one only if the local one is unhealthy.
• mTLS between clusters without touching the application.
• A single view of the traffic topology.

What they cost:

• A sidecar per pod (or eBPF, depending on the implementation), with its CPU and memory consumption.
• One more control plane to operate and upgrade.
• A whole layer of new failure modes, with a considerable learning curve.
• Notably harder debugging: traffic no longer goes where it appears to.

The honest criterion: a multi-cluster mesh is justified when there are many services calling each other across regions. If the communication between clusters comes down to database replication and a couple of calls, VPC peering and a few ExternalName Services solve the same thing at a fraction of the cost.

Rutas Norte does not use a service mesh. With two clusters, six components and a single cross-region conversation (PostgreSQL replication), it is not justified. It will be revisited if a third region appears.

This is where multi-cluster stops being a question of architecture and becomes a question of business survival.

The argument in favour of active-active is not the recovery time, it is that the recovery path is exercised every day. A passive secondary that has never served real traffic is a hypothesis, just like the backup that was never restored in 11-02.

The argument against is twofold: it costs twice as much, and above all it forces you to solve concurrent writes in two regions, which is an application design problem, not an infrastructure one.

The same concepts as in 11-02, now at the scale of a region:

The manifests replicate with git push. The application starts anywhere in minutes. The data is another matter.

Asynchronous PostgreSQL replication between regions. That is what Rutas Norte does: a replica in eu-west-3 following the eu-west-1 primary by streaming replication. The typical network latency between Ireland and Paris is 25-40 ms, which in practice means a replication lag of under a second under normal load, and a few seconds during the May bank-holiday weekend.

The consequence: if the main region disappears all at once, the committed transactions that had not yet reached the replica are lost. With a 15-minute RPO there is plenty of margin, but it is worth saying plainly: active-passive with asynchronous replication is not zero RPO. There can be bookings confirmed to the customer that do not exist after the failover, and the business has to know what to do about that.

Synchronous replication would give zero RPO, but every transaction commit would wait for a round trip to the other region: 35 ms added to every write. Over the May bank-holiday weekend, that is unaffordable.

And the even worse problem: coming back. When the main region recovers, its database is in a state divergent from the one that has been serving. You cannot simply switch back: you have to rebuild the old main region as a replica of the new one and then switch over cleanly. That procedure has to be written down before you need it.

The split solved the problem that motivated it: EKS upgrades are rehearsed in nonpro, and a webhook misconfigured by development can no longer affect ticket sales.

After the risk analysis, the board approved a secondary region in eu-west-3 (Paris) in active-passive mode, with these decisions:

The Argo CD point deserves attention. It is a classic mistake: putting the tool that deploys inside the cluster that may disappear. Rutas Norte solves it with a second instance in eu-west-3 pointing at the same repository, with syncPolicy.automated disabled under normal conditions; it is enabled as part of the failover procedure.

A procedure written down and never executed is an essay, not a plan. Rutas Norte runs a full annual drill in October, outside peak season:

Results of the October 2025 drill, the first one: a real RTO of 71 minutes against the 30-minute target. The findings were revealing and none of them would have surfaced without running it:

• The TLS certificate for eu-west-3 had expired: cert-manager could not complete the ACME challenge because DNS did not point to that region. It was fixed by using the DNS-01 challenge instead of HTTP-01.
• The ExternalSecrets in eu-west-3 pointed at a store that was not replicated to that region.
• Nobody could remember where the DNS failover procedure was written down.
• The PostgreSQL replica had not replicated for eleven days because of a firewall rule added in an unrelated change. Nobody knew, because there was no alert on cross-region replication lag.

That last finding, on its own, justified the entire drill: in a real disaster, eleven days of bookings would have been lost.

It is a serious investment. The question that justifies it is not technical: how much does one hour of downtime during the May bank-holiday weekend cost Rutas Norte? If it is €40,000 of lost sales plus the reputational damage, the investment pays for itself by avoiding a single regional incident. If the business can absorb half a day down without serious consequences, it does not.

• When one cluster still works. If namespaces, quotas and network policies cover your isolation needs, adding clusters only adds drift.
• When there is no GitOps. With manual deployments, N clusters mean N opportunities to diverge. GitOps is a prerequisite, not a complement.
• When there is no centralised observability. Diagnosing an incident by hopping between the dashboards of three clusters is unworkable. The logs and metrics of the whole fleet must be visible in one place.
• When the team cannot cope with one. Multiplying the infrastructure does not multiply the team's capacity, it divides it.
• When the reason is "just in case". With no concrete scenario to solve, you pay for the complexity and get nothing back.
• When the data cannot be replicated. If the application does not tolerate eventual consistency and there is no budget to redesign it, a second cluster gives a false sense of security.

Before considering a second cluster, it is worth having the earlier things sorted, because almost all of them improve reliability more for less money:

1. Tested backups and a timed restore (11-02).
2. GitOps with everything in Git (10-05).
3. Observability and alerts with runbooks (07-04, 11-06).
4. High availability within the cluster: multi-zone, PDB, topologySpread (09-05).
5. Safe deployments with a canary (11-04).
6. And then, yes, the second cluster.

A team that jumps to step 6 without the previous five ends up with two fragile clusters instead of one.

• Creating the second cluster with the same CIDR range as the first. It makes routing between them impossible and forces a rebuild. Plan the addressing for the whole fleet before creating the second.
• Putting Argo CD only inside the production cluster. When that cluster disappears, so does the ability to deploy anywhere. A secondary instance or a separate management cluster.
• Assuming DNS fails over instantly. Short TTLs are not always honoured; count on one to three minutes as a minimum.
• A secondary at zero replicas. You do not know whether a cold path works. Permanent minimum capacity and real traffic, even a little.
• Not alerting on cross-region replication lag. This is exactly the failure Rutas Norte discovered in the drill: eleven days without replicating and nobody knew.
• Duplicating the whole overlay when adding a region. Only what differs goes in the overlay; if it grows a lot, the base is badly factored.
• Installing a multi-cluster service mesh by default. It is one of the most complex pieces in the ecosystem. It is only justified with a lot of cross-region communication.
• Tip: put the context in your prompt today, with a red background for production. It is the highest-return-per-minute measure in the whole lesson.
• Tip: make production contexts read-only by default. Writing should require a deliberate, recorded act.
• Tip: write the failover procedure and the one for coming back. Coming back is harder than going, and almost nobody documents it.

A company has one cluster with dev, pre and pro in separate namespaces, GitOps with Argo CD, backups tested monthly and centralised observability. It has suffered two incidents in six months: one caused by an operator installed by development that consumed all the memory on a shared node, and another by a cluster version upgrade that left pro degraded for 25 minutes. The platform team is two people. Recommend a topology, justify it and state explicitly what you do NOT recommend and why.

Rutas Norte adds the rutasnorte-pro-w3 cluster with the labels environment=pro, region=eu-west-3 and role=secondary. Write an ApplicationSet that deploys bookings-api on every cluster with environment=pro, using the overlays/pro-{{region}} overlay, and that on clusters with role=secondary starts with fewer replicas. Also state what has to be true in the repository for it to work.

In the annual drill, a team gets: step 3 (PostgreSQL promotion) 4 min, step 4 (Argo CD and scaling) 22 min, step 5 (verification) 6 min, step 6 (DNS) 14 min. Total RTO 46 min against a 30-minute target. Identify the two problematic steps, propose a likely cause for each and a concrete corrective action.

Solution 1. Recommendation: topology by environment, two clusters — one for pro and another for dev+pre. Justification: the two incidents suffered are exactly the ones this topology solves; the first is resource and control-plane contention between environments, the second is the impossibility of rehearsing the upgrade before applying it to production. Besides, the prerequisites are covered: there is GitOps, tested backups and centralised observability, so drift is manageable. The incremental cost for two people is bearable because most of the operation is already automated.

What is not recommended: (a) one cluster per business unit, because with two platform people the proliferation is unaffordable and there is no problem motivating it; (b) a second region, because none of the incidents was regional and the cost (infrastructure plus data replication) is not justified by the available evidence; (c) a dedicated management cluster, unnecessary with only two clusters, where Argo CD can live in the production one managing both.

Solution 2.

Requirements in the repository: (a) the overlays/pro-eu-west-1 and overlays/pro-eu-west-3 directories must exist; (b) the reduced-capacity and full-capacity components must exist; (c) every target cluster must be registered in Argo CD with the environment, region and role labels; (d) the rutas-norte AppProject must have both servers in its list of permitted destinations, or Argo CD will reject the generated Applications. One additional note: since the HPA governs the replicas, the component should adjust the HPA's minReplicas/maxReplicas, not the Deployment's replicas field.

Solution 3. Problematic steps: 4 (22 min against an expected 5 or so) and 6 (14 min against 2).

Step 4: a likely cause is that the secondary region had no node capacity available and it was necessary to wait for the autoscaler to provision new nodes, including pulling images that were not cached on those nodes. Corrective action: keep permanent minimum capacity and pre-warm the images (for example, with a DaemonSet that pulls them, or by reserving nodes with the images already present). A complementary alternative: over-provision with low-priority pods that Karpenter evicts when scaling (09-03).

Step 6: fourteen minutes for a DNS change indicates it was not automated; somebody probably edited records by hand in a web console under pressure, or had to hunt for the credentials. Corrective action: script the change with the change file already written and versioned in the repository, runnable with a single command, and test it outside the drill. Also check that the records' TTL is 60 seconds and not the default value, which is usually much higher.

We have seen why a company ends up with several clusters — control-plane isolation, blast radius, latency across regions, data residency, differing versions and the simple fact that a cluster gets upgraded too — and what that really costs. We went through the common topologies with the recommendation to start with one and to split production from the rest first; the five layers of defence for never applying to the wrong cluster, of which the best value is putting the context in the prompt and the most effective is making production read-only by default; governing the fleet with the ApplicationSet cluster generator combined with Kustomize overlays; the idea of the cluster as a product to fight drift; the three levels of connectivity between clusters and the honest criterion for when a service mesh is justified; and disaster recovery, where the hard part is never the manifests but the data, with its asynchronous replication, its non-zero RPO and the return procedure that almost nobody writes.

The Rutas Norte plan, with two clusters today and a secondary region approved, illustrates the essential point: the decision is not taken for architectural elegance, but by comparing the cost of redundancy with the cost of an hour of downtime over the May bank-holiday weekend. And the October drill, with its real RTO of 71 minutes and its eleven days of broken replication that nobody had detected, leaves the most important lesson: a procedure that has not been executed is not a plan.

We now have the platform deployed, with state, with a pipeline, with safe deployments and with a governed fleet. What is missing is what no tutorial covers and what actually fills the working day: keeping it alive. In the last lesson of the module, Production Operations: Incidents, Runbooks and Costs, we will see how an incident is managed, how a useful runbook is written, how the error budget decides what the team does next month, how capacity for the May bank-holiday weekend is planned, and why the Kubernetes bill goes through the roof and what to do about it.