In the previous lesson we brought up disposable clusters on the laptop and discovered, almost in passing, that kind uses kubeadm internally: that kubeadmConfigPatches field in the configuration file was no accident. Now we go to the tool directly.

kubeadm is the Kubernetes project's official tool for bootstrapping a conformant cluster on machines you control. It is the foundation on which nearly every distribution is built, and understanding it gives you something no other route does: seeing with your own eyes where the certificates are, where the control plane's static pods live and exactly what happens when a node joins the cluster. Even if Rutas Norte S.L. ends up choosing a managed cluster (10-06), this knowledge is what will let you debug a broken control plane and pass the CKA certification (12-01).

1. What kubeadm is and what it is not
2. Self-managed versus managed: when it makes sense
3. Preparing the machines
4. Installing containerd
5. Installing kubeadm, kubelet and kubectl
6. kubeadm init with a configuration file
7. What kubeadm creates on the node exactly
8. Installing the CNI and why the nodes are NotReady
9. Joining worker nodes
10. A highly available control plane
11. Certificate management
12. Version upgrades
13. etcd backup and restore
14. kubeadm reset and clean-up
15. Common mistakes and tips
16. Exercises
17. Conclusion

kubeadm does one thing well: given machines that are already prepared, it bootstraps a control plane on them and joins worker nodes, producing a cluster that passes the Kubernetes conformance tests.

That right-hand list is the key. Plenty of people run kubeadm init, see the success message, and are then surprised that kubectl get nodes says NotReady. There is no error: the CNI is missing, and that job is not kubeadm's.

Before writing a single command, the honest question: should Rutas Norte S.L. operate its own cluster?

A serious warning: operating your own cluster in production demands a dedicated team. It is not a task you can bolt onto somebody else's job description. You need out-of-hours incident cover, a tested etcd restore procedure, an operating system patching policy, and someone who understands the certificates when they expire. If your organisation cannot sustain that, a managed cluster is not a convenience: it is a risk-management decision.

When kubeadm genuinely is the right answer: your own data centre or specific hardware (GPUs, low latency, regulatory compliance); a need for control plane configurations no managed offering allows; cost savings with many nodes; and learning and certification, because the CKA (12-01) assesses exactly this.

For Rutas Norte we will build a kubeadm cluster in a lab in order to learn and practise, with the reasoned production conclusion reserved for 10-06.

We start from three Ubuntu 24.04 LTS machines in the Rutas Norte lab:

Official minimum requirements: 2 CPUs and 2 GB of RAM per control node, a unique host name, a unique MAC address and a unique product_uuid (sudo cat /sys/class/dmi/id/product_uuid). If two cloned machines share a product_uuid, the cluster may confuse them: regenerate it from the hypervisor.

All the following steps are run on all three machines.

Why must swap be disabled? Because it breaks the resource model we studied in 03-04 and 03-05. The scheduler decides where a pod goes based on the memory requested and available; the kubelet evicts pods when the node is under memory pressure, following the QoS classes. With swap, a pod that exceeds its memory limit does not die: it starts paging to disk, becomes a thousand times slower, its liveness probes start failing erratically and the whole node degrades without any metric explaining it. It is a far worse failure than a clean OOMKilled.

Kubernetes 1.30 has experimental swap support (NodeSwap), but it is in beta with plenty of caveats. For a production cluster, swap off.

What each one does:

• overlay: the layered file system containerd uses to mount container images. Without it, the runtime cannot create containers.
• br_netfilter: makes traffic crossing a Linux network bridge (the one connecting the node's containers) visible to iptables. Without it, kube-proxy writes rules that are never applied to pod-to-pod traffic, and Services simply do not work.

• ip_forward = 1: the node must forward packets between interfaces. Without this, a pod on rn-worker-1 cannot talk to one on rn-worker-2.
• bridge-nf-call-iptables = 1: switches on what br_netfilter makes possible.

If sysctl tells you the parameter does not exist, it means br_netfilter is not loaded. The order matters.

About the last one: every CNI uses its own. Calico with VXLAN needs UDP 4789; with IP-in-IP it needs IP protocol 4; Cilium with VXLAN, UDP 8472. Check the documentation of the CNI you pick.

For the lab it is enough to open those ports with ufw and allow all internal network traffic: sudo ufw allow from 10.10.0.0/24. Verify from another node with nc -zv 10.10.0.11 6443.

Lab tip: if you are learning and something will not connect, temporarily disable the firewall (sudo ufw disable) to rule it out as the culprit. In production, never; there every rule is documented.

Kubernetes does not run containers directly: it talks to a runtime through the CRI interface. Since 1.24, Docker is not a valid runtime directly; containerd is the standard option.

Here is failure number one in kubeadm clusters. containerd ships by default with SystemdCgroup = false, whereas the kubelet in Kubernetes 1.30 uses the systemd cgroup driver. If they do not match, the cluster comes up apparently fine and then pods restart at random under memory pressure, because there are two cgroup managers fighting over the same node.

If the grep returns nothing or returns false, the option's path has changed in your version. Edit the file by hand and look for the [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options] section.

If this responds, the runtime is ready. crictl is your node-level debugging tool: crictl ps, crictl images, crictl logs. When the apiserver is down, kubectl is useless and crictl is all you have.

All three are installed from the official Kubernetes repository, which since 1.28 has been segmented by minor version. This matters: the v1.30 repository only contains 1.30 patches, which prevents accidental minor version jumps.

Why the hold is essential: without it, a routine system-maintenance apt upgrade would upgrade the kubelet on some random node. That new kubelet could be incompatible with the control plane, or simply restart mid-afternoon and evict every pod on that node. Kubernetes upgrades are a planned procedure (section 12), never a side effect.

You could bootstrap the cluster with a string of flags (kubeadm init --pod-network-cidr=... --control-plane-endpoint=... --apiserver-cert-extra-sans=...), and it would work. But the problem is obvious after everything we have covered so far: it is not versioned, nobody remembers which flags were used a year ago, and adding a control node means repeating them exactly. The professional way is a configuration file, kept alongside the manifests.

We run:

--upload-certs uploads the control plane certificates to a temporary, encrypted cluster Secret so that other control nodes can join without copying them by hand. It expires after 2 hours.

Output (heavily abridged):

Save that output. The two join commands at the end are what you need in sections 9 and 10.

NotReady. That is correct, and we explain it in section 8.

About controlPlaneEndpoint: even if today you have a single control node and no balancer, declare it with a DNS name from the outset. You can have that name point at 10.10.0.11 today and at the balancer's virtual IP tomorrow. If you do not, the certificate and every kubeconfig will point at the node's IP, and building HA later demands regenerating certificates across the whole cluster.

This section connects directly with the architecture we saw in 01-02. Now we are going to see it on disk.

This is a static pod: the kubelet watches that directory and starts any pod it finds there, without going through the apiserver. It is the answer to the chicken-and-egg problem: how does the apiserver start, if creating a pod requires an apiserver? Answer: it is not created as a normal pod, the kubelet reads it straight from disk.

Some very important practical consequences:

• If you edit /etc/kubernetes/manifests/kube-apiserver.yaml, the kubelet detects the change and restarts the apiserver automatically within seconds. That is how you add a flag by hand.
• If you delete that file, the apiserver vanishes and you lose the cluster. Take a backup before touching anything.
• kubectl delete pod kube-apiserver-rn-control-1 -n kube-system does not really remove it: the kubelet recreates it, because the source of truth is the file.
• If the apiserver will not start, kubectl does not work. Debug it with crictl ps -a and crictl logs, or with journalctl -u kubelet.

To inspect a certificate's valid names: sudo openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text | grep -A2 'Subject Alternative Name'.

In /etc/kubernetes/*.conf there is one per component: controller-manager.conf, scheduler.conf, kubelet.conf (this node's credentials, with automatic rotation) and two administrative ones:

• admin.conf: the one you copied to ~/.kube/config. Since 1.29 it belongs to the kubeadm:cluster-admins group, subject to RBAC (08-01).
• super-admin.conf: bypasses RBAC entirely (system:masters). It is the break-glass option when you have broken RBAC and cannot even fix it. Do not hand it around.

In /var/lib/etcd/member/ (with its snap and wal directories) lives the complete state of your cluster: every object, every Secret, everything. That is what has to be backed up (section 13).

The message is explicit. The kubelet refuses to declare itself Ready while there is no network plugin able to assign IPs to pods. Remember from 04-01: Kubernetes defines the contract (every pod has an IP, all pods see each other without NAT) but does not implement it. That is the CNI's job.

You will also see CoreDNS in Pending, and that is normal: CoreDNS is a pod and it needs the network to start.

We install Calico (the CNI detail is in 04-01; here only the essentials):

And CoreDNS starts on its own. The full sequence is:

On rn-worker-1 and rn-worker-2, already prepared per sections 3, 4 and 5:

What the two parameters mean:

• --token: a temporary credential (it expires after 24 hours) authorising the node to request its certificate. It is a Secret of type bootstrap.kubernetes.io/token in kube-system.
• --discovery-token-ca-cert-hash: the SHA-256 hash of the public key of the cluster's CA. Its purpose is for the node to verify the apiserver, not the other way round. Without it, an attacker could impersonate the control plane and capture the token. Never use --discovery-token-unsafe-skip-ca-verification outside a lab.

This is the most common situation: you want to add a node three weeks later and the original token is no longer valid.

Related commands:

kubeadm does not set the role label on worker nodes for security reasons (a kubelet cannot assign itself role labels). It is done from the control plane:

With a single control node, if rn-control-1 goes down the cluster keeps serving traffic (the pods keep running, kube-proxy keeps routing) but you lose all management capability: no pod rescheduling, no scaling, no deployments, no HPA. And if etcd's disk is lost, you lose the entire cluster.

Every control node publishes the same port 6443. You need something in front to spread the load:

Options: HAProxy + keepalived with a floating virtual IP (the most common choice in your own data centre), the cloud layer-4 balancer if you are on virtual machines, or kube-vip as a static pod if you want no external infrastructure. What does not work is DNS with several A records: the client caches and does not notice failures.

A minimal HAProxy configuration:

It is pure TCP (tcp mode), not HTTP: the connection to the apiserver is end-to-end TLS and the balancer must not terminate it. With check, if a node stops answering on 6443, HAProxy automatically stops sending it traffic.

etcd needs quorum: 1 member tolerates 0 failures, 2 members also tolerate 0 (worse than 1!), 3 tolerate 1, 4 tolerate 1, and 5 tolerate 2. That is why the number of control nodes is always odd: 1 (lab), 3 (normal production) or 5 (large clusters).

For external etcd, you declare it in the kubeadm file:

The --certificate-key comes from --upload-certs. If more than 2 hours have passed, regenerate it:

To check the health of the etcd cluster, kubectl -n kube-system exec etcd-rn-control-1 -- etcdctl ... endpoint status --cluster -w table must show all three members, with the same version, a similar database size and exactly one with IS LEADER = true.

Here is the trap that catches out most teams: the certificates kubeadm generates expire after a year. A cluster built in March stops working in March the following year, with no warning, with a message like this:

Kubernetes renews the certificates automatically during kubeadm upgrade. Since many teams upgrade at least once a year, they never see it. Whoever does not upgrade gets the fright.

CAs last 10 years; the certificates they sign, 1 year.

Operational recommendation: do not leave this to a human calendar. The apiserver exposes apiserver_client_certificate_expiration_seconds_bucket, on which you can build an alert in Alertmanager (07-04). Simpler and even more reliable: a CronJob (06-03) or a system task running kubeadm certs check-expiration weekly and warning if less than a month remains.

A note about the kubelet: its certificate rotates on its own, automatically, if rotateCertificates: true (the default). The certificates you have to renew by hand are the control plane's.

You can only jump one minor version at a time. You cannot go straight from 1.28 to 1.30: you have to do 1.28 → 1.29 → 1.30. Each jump is a complete procedure.

In addition, the version skew policy requires:

Hence the mandatory order: the control plane first, the worker nodes afterwards. Never the other way round.

Three things, in this order: an etcd backup (section 13, non-negotiable), reading the release notes, and checking that nothing uses APIs that are going away.

Any line here is work that has to be done before upgrading. Tools such as kubent (kube-no-trouble) or pluto scan your manifests looking for the same thing.

For the other control nodes and for the worker nodes, the procedure is the one from the previous section with a single difference: you use sudo kubeadm upgrade node instead of upgrade apply.

One at a time, verifying in between. If drain hangs, that is a PodDisruptionBudget doing its job: bookings-postgres cannot be left without replicas. Investigate with kubectl get pdb -A before forcing anything.

In 05-06 we backed up the applications' data with Velero. This is a different thing: it is a backup of the cluster's definition. Every Deployment, Secret, ConfigMap, RBAC rule, CRD... it all lives in etcd.

Without an etcd backup, a disk failure on the control node forces you to recreate the cluster from scratch and reapply every manifest. With GitOps (10-05) that would be recoverable, but you would lose everything that is not in Git.

That same command in /etc/cron.d/etcd-backup, at 03:15 daily and chained with a find /var/backups -name 'etcd-*.db' -mtime +14 -delete, covers automation with a two-week retention.

Three essentials: also back up /etc/kubernetes/pki/ (a snapshot without the CA is useless), take them off the node (a backup on the disk that is going to fail is not a backup) and encrypt them, because they contain every Secret in rutas-norte-pro, including the bookings-postgres credentials and those of the payment gateway: remember from 03-02 that Secrets are base64-encoded, not encrypted.

An emergency procedure, for when the control plane is unrecoverable:

Critical points:

• On an HA cluster, you have to restore on all three control nodes, with the identical snapshot and each one with its own --name and URLs. If the data differs, etcd will not form quorum.
• The state goes back to the moment of the backup. Everything created afterwards is lost.
• Pods that were running keep running (the kubelet has not noticed), but "ghost" objects can appear: pods that exist on the node but not in etcd, or the other way round. After restoring, check kubectl get pods -A against crictl ps on every node.

Rehearse the restore. A backup that has never been restored is not a backup: it is a file. Do it in the lab, time yourself, and write the procedure down in a runbook (11-06).

When something goes wrong and you want to start over, or to retire a node:

reset is honest about what it does not clean up. You have to finish the job by hand:

If you skip steps 3, 4 and 5, the next kubeadm init or join on that machine may appear to work and then produce inexplicable network failures: stale iptables rules routing to pod IPs that no longer exist. It is one of the most frequent causes of "I reinstalled and it still fails".

1. SystemdCgroup = false in containerd. This is mistake number one. The cluster starts, everything looks fine, and days later pods restart under memory pressure with no explanation. Always verify with grep SystemdCgroup /etc/containerd/config.toml and restart containerd after changing it.

2. Forgetting swapoff -a in /etc/fstab. You disable it, it works, you reboot the machine a month later and the kubelet will not start. Always comment out the line in /etc/fstab.

3. A podSubnet that overlaps the physical network. If your lab uses 10.10.0.0/24 and you declare podSubnet: 10.0.0.0/8, routing breaks in ways that look like black magic. Pick ranges that collide with nothing: not the node network, not serviceSubnet, not the corporate network.

4. A CNI range different from podSubnet. You declare 10.244.0.0/16 in kubeadm and install Calico with its default value 192.168.0.0/16. Pods get IPs from the CNI's range, but the controller-manager allocates blocks from the other one. They must match.

5. Not declaring controlPlaneEndpoint from the outset. Adding high availability later forces a certificate regeneration across the whole cluster. Declare a DNS name from day one, even if it points at a single IP.

6. Certificates expiring after a year. Real and very common. Put an alert in the calendar, or better, in your monitoring.

7. Jumping two minor versions, or upgrading the nodes before the control plane. kubeadm upgrade apply v1.32.0 from 1.30 is rejected; and a 1.31 kubelet against a 1.30 apiserver is outside the skew policy. One minor at a time, control plane first.

8. kubectl drain hanging and forcing it with --force. What is blocking you is usually a PDB (09-05) protecting bookings-postgres or redis-cache. Forcing it can cause data loss. Investigate first with kubectl get pdb -A.

9. etcd backups without /etc/kubernetes/pki. A snapshot without the CA will not give you back a working cluster. Back up both, together, encrypted and off the node.

10. kubeadm reset without cleaning iptables and the CNI. Leftovers cause network failures on reinstall. Run the full clean-up from section 14.

11. Not setting apt-mark hold. A routine apt upgrade that upgrades the kubelet in production is an incident waiting to happen.

Write an idempotent prepare-node.sh script (one that can be run twice without breaking anything) that leaves an Ubuntu machine ready for kubeadm join: swap permanently disabled, modules and sysctl, containerd with a verified SystemdCgroup=true, and kubeadm/kubelet/kubectl 1.30.4 pinned. The script must fail with a clear message if any check does not pass.

Write the kubeadm configuration file for the rutas-norte-lab cluster with these requirements: three control nodes behind api-k8s.rutasnorte.example (virtual IP 10.10.0.10), podSubnet 172.16.0.0/16, stacked etcd, auditing enabled, and the kubelet reserving 500m of CPU and 512Mi of memory for the system. Explain why you chose 172.16.0.0/16 and what you would check beforehand.

Write out the complete procedure, in order and with the exact commands, to upgrade the cluster from 1.30.4 to 1.31.1 with one control node and two workers, including the prior backup. Add the "point of no return" criterion and what you would do if the apiserver will not start after upgrade apply.

Solution 1

The keys to idempotency: the sed that does not re-comment already-commented lines, the [[ -s ]] before regenerating containerd's configuration, and the unhold before installing.

Solution 2

Why 172.16.0.0/16: it is a private range (RFC 1918) that collides neither with the lab network (10.10.0.0/24) nor with serviceSubnet (10.96.0.0/12), and it gives 65,536 IPs, which is plenty. Prior checks: that 172.16/16 is not used on the corporate network or the VPN, and configuring the CNI with that same CIDR.

Solution 3

Point of no return: kubeadm upgrade apply, because it migrates etcd's schema. Before it, reverting the packages is enough; after it, going back demands restoring the snapshot.

If the apiserver will not start: sudo crictl ps -a | grep apiserver, sudo crictl logs <id> and journalctl -u kubelet -f. Typical causes: an unsupported new flag or a badly generated certificate. If it is not resolved within the maintenance window, restore pre-131.db and pki-pre-131.tgz using the procedure in section 13.2 and revert the packages to 1.30.4.

You now know how to build a Kubernetes cluster from scratch on your own machines. The essentials:

• kubeadm bootstraps a conformant cluster; it does not provision machines and does not install the CNI. That boundary explains 90% of the initial confusion, including the NotReady node.
• Preparing the machines — swap, overlay and br_netfilter, ip_forward, ports, and above all SystemdCgroup = true in containerd — is where the hardest-to-diagnose failures are born.
• A versioned configuration file (ClusterConfiguration, InitConfiguration, KubeletConfiguration) is infinitely better than a string of flags; controlPlaneEndpoint and certSANs set correctly from day one save you regenerating certificates later.
• kubeadm leaves on the node static pods in /etc/kubernetes/manifests, the certificates in /etc/kubernetes/pki and the kubeconfigs: the architecture from 01-02, turned into files.
• Nodes join with a token and the CA hash; the token expires after 24 hours and is regenerated with kubeadm token create --print-join-command. High availability demands a stable entry point and an odd number of etcd members, normally 3 stacked.
• Certificates expire after a year, upgrades go one minor version at a time with the control plane before the worker nodes, and etcd backups (snapshot + pki, encrypted, off the node, with a rehearsed restore) are the difference between a bad day and the end of the company.

And the warning that bears repeating: operating this in production demands a dedicated team with on-call duty. In 10-06 we will see what a managed cluster saves you and at what price.

But first we have to solve the problem this module started with. We now know how to create clusters, both local and our own; what remains unsolved is Rutas Norte's 120 duplicated YAML files. In the next lesson, Helm, we will meet the Kubernetes package manager: charts, values, templates and releases. It is the tool we already used to install cert-manager (04-05) and kube-prometheus-stack (07-03) without explaining it; time to understand it properly and use it to package the Rutas Norte platform into a single chart with one values file per environment.