In 10-02 we built a cluster with kubeadm and discovered everything it entails: preparing machines, configuring containerd, generating certificates that expire after a year, maintaining etcd's quorum, rehearsing restores and upgrading node by node. We ended with a clear warning: operating that in production demands a dedicated team with on-call duty.

In 10-05 we set up GitOps, and now the entire state of the Rutas Norte platform lives in Git. That raises a natural question: if the cluster is replaceable and rebuilds itself, why would we maintain it ourselves?

This lesson answers that question. We will look at what the provider manages and what remains yours, we will compare EKS, AKS and GKE on the criteria that genuinely decide things, we will close the identity federation left pending in 03-06, we will take advantage of spot instances for Rutas Norte's tolerant workloads, and we will finish with a reasoned recommendation.

1. The shared responsibility model
2. Comparing EKS, AKS and GKE
3. Identity: IRSA and Workload Identity
4. Nodes: managed groups, templates and spot instances
5. What changes in what you already know
6. Upgrading a managed cluster's version
7. Real cost and the levers for saving
8. Portability and vendor lock-in
9. The decision for Rutas Norte
10. Common mistakes and tips
11. Exercises
12. Conclusion

A managed cluster is not "Kubernetes without work". It is a different boundary between what the provider does and what you do.

The control plane disappears from your worries. The certificates expiring after a year, etcd's quorum, the snapshots, the HAProxy in front of the three control nodes: all of that goes away. That is roughly half of lesson 10-02.

Everything else stays the same. The nodes are still yours, with an operating system that has to be patched. RBAC is still yours. The NetworkPolicies are still yours. And if bookings-postgres loses data, that is your problem, not the provider's.

A very frequent mistake: "it is managed, so backups are taken care of". False. The provider backs up etcd (the objects' definitions), not your applications' volumes. You back up bookings-postgres's customer data yourself, with Velero or the provider's snapshots. We saw this in 05-06 and it still holds.

The golden rule: a managed cluster takes half the Kubernetes operations work off your hands. The other half, the half you studied in modules 2 to 9, remains entirely yours.

With three environments in separate clusters, that is around $220/month just for existing. It is one of the reasons why many companies separate environments by namespace within one cluster (as Rutas Norte does) rather than by cluster.

Warning: AKS's free tier has no financial SLA. For rutas-norte-pro that may be unacceptable, and then the cost evens out.

GKE's channel system is a real operational difference: you choose "stable" and Google keeps the cluster on a proven version, upgrading it in the maintenance window you specify. It is the closest thing to never thinking about versions.

In EKS and AKS, the cycle is more manual. And if you let the version lapse, both end up forcing the upgrade, which is worse than planning it.

This is the most important and least known technical difference, and the one that causes serious surprises in production.

The problem this causes in practice: on EKS, every pod consumes a real IP from your VPC. A t3.medium instance supports 3 network interfaces with 6 IPs each: 17 pods maximum, regardless of having free CPU and memory. If your subnet is a /24 (254 addresses), you run out of IPs for pods long before you run out of CPU.

And the symptom when they run out:

A Pending pod with no shortage of CPU or memory. Baffling if you do not know the model.

Mitigations on EKS: enable prefix delegation mode (ENABLE_PREFIX_DELEGATION), which multiplies the available IPs per interface by 16; use larger instances; or plan generous VPC ranges from the outset (a /16, not a /24).

On AKS with classic Azure CNI, you have to reserve in advance maxPods × number of nodes addresses. Plan badly and you cannot scale. The overlay variant (Azure CNI Overlay) solves this and is the recommended option today.

On GKE with VPC-native, secondary alias IP ranges are used, which separates pod addressing from node addressing. It is the cleanest model of the three.

All three have converged on the same mechanism: OIDC. We go into it in section 3.

GKE Autopilot deserves a mention of its own: it is a mode in which, from your point of view, there are no nodes at all. You declare pods with their requests and you pay for exactly that. There are no nodes to patch or to size, no DaemonSets of your own, no hostPath and no privileges. It is Kubernetes reduced to its workload interface.

• In favour: zero node operations, exact billing for what you request, hardened security by default.
• Against: strong restrictions (you cannot run Falco from 08-06 as is, nor certain DaemonSets), a higher unit cost if your pods are badly sized, less control.

AWS Fargate is similar but more limited: it does not support DaemonSets at all, nor persistent block storage, nor privileged. For notifications-worker it can fit; for bookings-postgres, no.

Recalling 09-03: Karpenter does not work with predefined node groups, it picks the optimal instance type for the pending pods. If occupancy-reports needs 8 CPUs once a night, Karpenter brings up exactly the right instance, uses it and retires it. It is a notable difference in cost and in speed.

The honest conclusion: the best choice is nearly always the cloud where the rest of your infrastructure already lives. The technical difference between the three is smaller than the cost of operating in two clouds at once. If you are starting from scratch and only looking at Kubernetes, GKE is the most polished; if your company already lives in AWS, EKS with Karpenter is excellent; if your company is a Microsoft shop, AKS integrates without friction.

Here we close what we flagged in 03-06.

bookings-api needs to read a fares file from a storage bucket and publish messages onto a queue. The traditional way:

Problems: the credential is permanent, it has to be rotated by hand, it lives in a Secret (base64, not encrypted, 03-02), and if it leaks it works from anywhere in the world.

All three providers converge on the same mechanism. The cluster exposes an OIDC issuer; the kubelet projects a short-lived signed token carrying the ServiceAccount's identity into the pod; the cloud provider trusts that issuer and exchanges the token for temporary credentials.

No permanent credential. Nothing to rotate. The token renews itself and only works for that ServiceAccount in that cluster.

The magic is in the role's trust policy, whose condition limits who can assume it:

Note the sub: only the bookings-api ServiceAccount in the rutas-norte-pro namespace can assume this role. Not another SA, and not the same SA in another namespace.

In the Deployment, serviceAccountName: bookings-api is enough: no credential variables, because the provider's SDK detects the projected token automatically. You can verify it with kubectl exec ... -- ls /var/run/secrets/eks.amazonaws.com/serviceaccount/.

Note: AWS has introduced EKS Pod Identity, which is simpler (an association in the EKS API, with no OIDC provider and no annotation). It is the recommended option for new clusters; IRSA is still needed if the workload runs outside EKS.

The mechanism is the same; the commands and the annotation change.

What matters for you: in all three cases, the manifest that goes into Git contains an annotation with an identifier, never a secret. It is exactly the model we were after in 03-06 and it fits perfectly with GitOps (10-05).

A managed node group is a set of homogeneous machines the provider maintains: it creates them, joins them to the cluster, upgrades them gradually and replaces them if they fail.

Note the instanceTypes with four types in the spot group: the more types you accept, the lower the chance that the provider runs out of capacity and interrupts everything at once.

They are the provider's spare capacity at a 60-90% discount, with a trade-off: the provider can reclaim it with a 2-minute warning (AWS), 30 seconds (Azure) or 30 seconds (GCP).

The manifests, using what we learned in 06-05 and 09-05 (the full detail is in the solution to exercise 1):

And a PDB with minAvailable: 1, because even though they are interruptible, we do not want them all going down at once (09-05).

An important nuance about the PDB: a PodDisruptionBudget protects against voluntary disruptions (draining a node to upgrade it). A spot instance interruption is involuntary: the provider takes the machine away, PDB or no PDB. The PDB helps because the termination handlers (the AWS Node Termination Handler, or Karpenter's native handling) cordon and drain the node on receiving the warning, and that drain does respect the PDB.

occupancy-reports, being a nightly CronJob (06-03), is an even clearer case: on top of the toleration it can carry a hard nodeSelector on rutasnorte.example/spot: "true" — if there is no capacity, waiting is acceptable — and a generous backoffLimit: 6, because interruption is to be expected.

Instead of defining groups with fixed types, you declare constraints and Karpenter chooses:

Differences from the Cluster Autoscaler (09-03):

This section revisits the course's earlier lessons and points out what is different in the cloud.

In minikube we used standard, a node directory. In the cloud:

The classic mistake: bookings-postgres with volumeBindingMode: Immediate creates the disk in eu-west-1a; the scheduler places the pod in eu-west-1c; the pod stays Pending for ever with volume node affinity conflict. The fix: WaitForFirstConsumer, which is the default in all three providers' StorageClasses.

Locally it stayed in <pending> and needed minikube tunnel. In the cloud it genuinely works:

Things to learn:

• Every LoadBalancer Service costs money (around $20-25/month plus traffic). That is why only one is used for the Ingress controller and everything else goes through Ingress rules (04-04).
• The annotations are provider-specific. They are the least portable part of your manifests (section 8).
• Deleting the Service deletes the balancer. And if the balancer has a DNS record pointing at it, that breaks.
• externalTrafficPolicy: Local preserves the client IP but requires a pod with endpoints on every node; otherwise traffic to that node is dropped.

All three providers additionally offer a managed Ingress controller (AWS Load Balancer Controller, Application Gateway Ingress Controller, GKE Ingress) that creates a layer 7 balancer straight from the Ingress object. It is convenient, but it ties you to the provider more than ingress-nginx does.

The practical advantage: imagePullSecrets disappears. The node authenticates with its cloud identity. One less Secret to manage.

And it dovetails with 08-05: signing with Cosign and the admission policy work the same, with the bonus that all three registries offer automatic scanning that feeds into what we learned in 08-06.

Pragmatic advice: keep Prometheus (07-03) for the applications' metrics, because your alerts and dashboards are portable, and use the provider's service for the logs (07-05), because operating EFK at scale is expensive and painful, and the managed service's cost usually pays off. It is the combination most teams choose.

Beware of log costs: bookings-api with LOG_LEVEL=debug in production can generate hundreds of GB per month. All three providers charge for ingestion.

Compared with the kubeadm procedure (10-02), this is a walk in the park. But it still demands planning, and whoever treats it as a button gets an unpleasant surprise.

1. Removed APIs are still your problem. If a manifest in k8s/base uses an API that disappears in 1.31, the cluster upgrades and your application stops deploying.

2. Third-party add-ons have their own compatibility matrices. cert-manager, ingress-nginx, KEDA, the Prometheus Operator, the CSI driver: each supports a range of versions. Upgrading the cluster without upgrading them can break them.

3. Upgrading the nodes restarts every pod. The provider drains each node in turn. That exercises:

• Your PodDisruptionBudgets (09-05): if they are misconfigured, the drain hangs or, worse, gets skipped with --force.
• Your readiness probes (07-01): if bookings-api takes 45 seconds to become ready and the drain moves fast, there is an outage.
• Your terminationGracePeriodSeconds and your preStop hook.
• Your capacity to absorb the temporary loss of a node: if the cluster is 95% full, draining a node leaves pods Pending.

4. The control plane and the nodes are upgraded separately, and you have to respect the version skew (10-02): nodes at most 3 minors below the apiserver, and never above.

That last point closes the circle with the module's first lesson: the local environment must follow production.

A common surprise: traffic between availability zones is billed. If bookings-api is in zone A and bookings-postgres in zone B, every query crosses a billed boundary. At high volume, that is hundreds of euros a month. It is an argument in favour of zone affinity... which clashes with the high availability of 09-05. It is a real trade-off that has to be decided deliberately.

1. Sizing properly (what we learned in 09-02). It is the lever with the best effort-to-saving ratio, and hardly anybody uses it. The typical pattern: somebody set requests: {cpu: 1, memory: 2Gi} "just in case" and the pod uses 80m and 200Mi. The scheduler reserves what you asked for, so you are paying 12 times what you use.

Adjusting requests to the recommendation across the whole platform usually cuts the compute bill by between 30 and 50%.

2. Real autoscaling (09-01 and 09-03). With the HPA adjusting pods and Karpenter or the Cluster Autoscaler adjusting nodes, you pay for the real load and not for the annual peak. Rutas Norte has peaks at bank holidays: without autoscaling, you either size for the bank holiday all year round, or you fall over on the bank holiday.

3. Spot instances. A 60-90% discount on notifications-worker and occupancy-reports.

4. Switching off environments nobody uses at night. rutas-norte-dev does nothing between 20:00 and 08:00 or at weekends. That is 128 out of 168 weekly hours: a 76% saving in that environment.

Careful with GitOps: if Argo CD has selfHeal on in rutas-norte-dev, it will undo the scale to zero within three minutes. That is why in the solution to exercise 1 of 10-05 we left selfHeal: false in dev. The clean alternative is to scale the node group instead of the Deployments, or to use an AppProject sync window.

5. Usage commitments. All three providers discount 30-55% in exchange for committing to spend or capacity for 1 or 3 years (Savings Plans and Reserved Instances on AWS, Reservations on Azure, Committed Use Discounts on GCP).

The recommended strategy: commit only the stable baseline (the nodes that are always on), leave the peak on demand and the tolerant workloads on spot.

6. Cost visibility per team. Tools such as OpenCost or Kubecost break the spend down by namespace, label or team. Without that, nobody knows that occupancy-reports costs €400/month and nobody has any incentive to fix it. The labels from 02-07 (app.kubernetes.io/part-of, environment) are the basis of that breakdown.

All of this counts for little if in two years you want to change cloud and cannot. Let us be precise about what is portable at Rutas Norte.

The obvious pattern: what is inside Kubernetes is portable; what is outside is not. Vendor lock-in does not come from EKS, AKS or GKE, but from the services surrounding the cluster.

1. Isolate the non-portable parts in Kustomize overlays. You already have this set up from 10-04:

Changing cloud means changing one line in components:. The base is not touched.

2. Use ingress-nginx rather than the provider's controller. It is a layer of abstraction that costs a little performance and saves an entire migration. Ingress rules and nginx annotations work the same in all three clouds.

3. Use Prometheus for the metrics. Your alerts and dashboards are years of accumulated operational knowledge. In PromQL they are portable; in the provider's query language, they are not.

4. Terraform rather than the native tool. It does not make the infrastructure portable, but it does make the mental model and the workflows portable.

5. Decide deliberately about the databases. Running bookings-postgres as a StatefulSet with an operator (06-01, 06-07) is portable; using the provider's managed service is more convenient and less portable. Both are defensible decisions; what is not acceptable is making the decision without noticing.

6. Be realistic. Total portability costs money and complexity, and many companies never change cloud. A sensible goal: "we could migrate in three months with effort, not in three days". That gives you negotiating power and protects you from an abusive price change, without paying the tax of total abstraction.

• A small company: two people in platform, six in development, three in support.
• No 24×7 on-call cover.
• Traffic with strong, predictable peaks: bank holidays and holiday seasons, 8-10 times the base load.
• Customers' personal data in bookings-postgres: GDPR requirements on location and encryption.
• A tight budget.
• It already uses AWS object storage and transactional email for other things.

With two people in platform and no on-call rota, kubeadm (10-02) is ruled out. The decisive question is: who restores etcd at 4 in the morning on a Sunday? If the answer is "we wait until Monday", you cannot operate rutas-norte-pro on a cluster of your own. The control plane saving (about €220/month with three clusters) does not come close to offsetting the risk.

If Rutas Norte were not already using AWS, the recommendation would be GKE: the cleanest networking model, simpler Workload Identity, release channels that reduce upgrade work, and Autopilot as an option for the non-production environments.

Two clusters, not three. Separating pro from the rest is a real security boundary; separating dev from pre does not justify it: namespaces with quotas (03-04) and NetworkPolicies (04-06) are enough, and they save $73/month plus the operational work.

Without the saving levers (everything on demand, no spot, no night-time shutdown, no commitments, with oversized requests) the same platform would cost in the order of $1,400-1,600/month. The levers in section 7 are not theory: they are more than half the bill.

A managed cluster is not cheaper than your own in pure infrastructure cost. It is cheaper in total cost, because that includes the work you do not have to do and the risk you do not take on. For Rutas Norte S.L., with two people in platform and no on-call rota, it is the only defensible option.

And there is a symmetry worth noting: thanks to GitOps (10-05), the decision is reversible. The entire state of the platform is in Git. If tomorrow there is a reason to change cloud or to build a cluster of our own, you bring up the new cluster, install Argo CD, apply the root Application and twenty minutes later the platform is standing. The data migrates separately, which is the hard part, but the platform rebuilds itself.

1. Believing that "managed" includes backups of your data. The provider backs up etcd, not your volumes. bookings-postgres is your responsibility: Velero (05-06) and snapshots, with a rehearsed restore.

2. Not planning the IP addressing. On EKS with VPC CNI, every pod consumes a real VPC IP. A /24 runs out quickly and the symptom (FailedCreatePodSandBox) does not mention IPs obviously. Plan generous ranges and enable prefix delegation.

3. One LoadBalancer Service per application. Each one costs $20-25/month. Use one for the Ingress controller and route with Ingress rules (04-04).

4. A StorageClass with volumeBindingMode: Immediate in a multi-zone cluster. The disk is created in one zone, the pod is scheduled in another, and the pod never starts. Use WaitForFirstConsumer.

5. Putting stateful workloads on spot instances. bookings-postgres on spot is a recipe for losing data. Only workloads that tolerate dying without warning, with an appropriate terminationGracePeriodSeconds and preStop hook.

6. A single instance type in the spot group. If the provider runs out of that type, it takes all your nodes at once. Declare 4-6 equivalent types.

7. Carrying on with static credentials in Secrets. IRSA, Workload Identity and their Azure equivalent eliminate permanent credentials. It is one of the best cost-to-benefit security improvements there is.

8. Upgrading the version without checking the removed APIs or the add-ons. The cluster upgrades without a hitch and your application stops deploying, or ingress-nginx stops starting. kubent and the compatibility matrices, before touching anything.

9. Oversized requests "just in case". It is the leading cause of overspend in Kubernetes. The VPA in Off mode (09-02) gives you the real numbers within a week.

10. Ignoring cross-zone traffic. In very chatty applications it can be a significant line item. Measure it before deciding; it is a real trade-off against the high availability of 09-05.

11. Not labelling for cost allocation. Without environment, app.kubernetes.io/part-of and a team label, the bill is one global number nobody can reduce. The labels from 02-07 are the basis of OpenCost or Kubecost.

12. Adopting Autopilot or Fargate without checking the restrictions. Falco (08-06), some networking DaemonSets and anything requiring privileges may not work. Try it in a non-production environment first.

13. Confusing "less operations" with "no operations". You are still responsible for the nodes, RBAC, policies, secrets, observability and cost. Modules 2 to 9 of this course remain entirely applicable.

Write the complete set of manifests that lets notifications-worker run on a spot node group marked with the taint rutasnorte.example/spot=true:NoSchedule, while still working if there is no spot capacity. It must include: tolerations, preferred node affinity, topology spread, terminationGracePeriodSeconds with a preStop hook, a PDB and a KEDA ScaledObject. Explain why you use preferred affinity and not a nodeSelector, and what the PDB really protects against in a spot interruption.

Design the k8s/components/provider-aws/ component containing everything AWS-specific that Rutas Norte needs: the rutas-norte-ssd StorageClass with encrypted EBS gp3, a patch adding the IRSA annotation to the bookings-api and notifications-worker ServiceAccounts, and a patch adding the network balancer annotations to the Ingress controller's Service. Explain how it would be enabled from the pro overlay and what would have to change to migrate to Azure.

The rutas-norte-pro bill is $1,450/month: $980 of compute, $180 of storage, $120 of load balancers (5 LoadBalancer Services), $90 of traffic and $80 of logs. Every pod has requests: {cpu: 500m, memory: 1Gi} and kubectl top shows an average usage of 90m and 220Mi. There are no spot instances and no commitments. Propose a prioritised plan with the diagnostic commands, the estimated saving of each measure and the associated risk.

Solution 1

Preferred affinity and not a nodeSelector: nodeSelector is a hard constraint. If there is no spot capacity in the region — which happens precisely at peak times, when you need it most — the pods would stay Pending indefinitely. With preferredDuringScheduling, the scheduler tries the spot node and, failing that, uses an on-demand one: you pay more, but the service does not stop.

What the PDB protects: nothing against the interruption itself (the provider takes the machine). It protects during the drain that the termination handler (or Karpenter) runs on receiving the 2-minute warning: that drain respects the PDB, so two nodes are not emptied simultaneously leaving the queue without consumers.

Solution 2

storageclass.yaml is the one from section 5 (provisioner ebs.csi.aws.com, encrypted gp3, WaitForFirstConsumer). The two patches are minimal strategic merges; the target takes care of who they apply to, so the patch's metadata.name is irrelevant:

Migrating to Azure: create k8s/components/provider-azure/ with provisioner: disk.csi.azure.com, the azure.workload.identity/client-id annotation (plus the azure.workload.identity/use: "true" label on the pods) and Azure's balancer annotations. Then change that single line. k8s/base/ is not touched at all.

Solution 3

Order and risk: you start with the requests adjustment because it has the greatest saving and the lowest risk, applying it first in pre, verifying with k6 (09-06) and then in pro component by component. The usage commitment goes last on purpose: committing to a capacity only makes sense after you have reduced it, or you would be paying upfront for your own oversizing.

Module 10 ends here. The essentials of this lesson:

• The shared responsibility model takes the entire control plane off your hands — etcd, certificates, quorum, upgrades, high availability — which is roughly half the work of 10-02. Everything else, including the nodes, RBAC, policies, secrets, observability, backups of your data and cost, remains yours.
• EKS, AKS and GKE are more alike than different. The differences that genuinely decide things are the networking model and pod IP allocation, the version cycle, the autoscaling (Karpenter stands out) and the automatic modes (Autopilot stands out). And above all: the cloud where your infrastructure already lives.
• Identity federation — IRSA, Workload Identity and their Azure equivalent — eliminates permanent credentials from Secrets. The manifest that goes into Git carries an annotation, not a secret. It closes what we promised in 03-06.
• Spot instances are a real saving for notifications-worker and occupancy-reports, provided they come with tolerations, preferred affinity, topology spread, preStop and a PDB.
• What you already knew changes in concrete ways: the StorageClass is the provider's and needs WaitForFirstConsumer, the LoadBalancer finally gets an IP (and a bill), the registry authenticates with the node's identity, and metrics and logs can be delegated.
• Upgrading still demands planning: removed APIs, add-on compatibility matrices, PDBs that allow draining and probes that survive the restart.
• Cost is mastered with six levers, and the first two — adjusting requests with the VPA and switching off what nobody uses — have the best effort-to-benefit ratio.
• And portability depends on what you put outside Kubernetes, not on which managed offering you choose. Isolating the provider-specific parts in a Kustomize component leaves the base 100% portable.

For Rutas Norte S.L., the recommendation is EKS in eu-west-1, with two clusters, Karpenter, spot instances for the tolerant workloads and the saving levers applied from day one. And thanks to GitOps, that decision is reversible.

This brings the tour of the ecosystem to a close. You now have all the pieces: local and self-built clusters, packaging with Helm, overlays with Kustomize, continuous deployment with GitOps and a managed platform to rest it all on. Added to the nine previous modules — pods, services, configuration, networking, storage, advanced patterns, observability, security and scaling — you now know all the pieces individually.

What is missing is seeing them work together. In Module 11: Case Studies and Real-World Applications we will walk through complete scenarios from start to finish: deploying a web application from scratch, running stateful applications, a complete CI/CD pipeline, blue-green and canary deployment strategies, multi-cluster management and, to close, real production operations: incidents, runbooks and cost control. That is where knowledge turns into judgement.