Let us begin with what no Kubernetes lesson usually says in its first paragraph: for Tramontana S.L., Kubernetes is disproportionate. One application, one database, three people and five hundred bookings a month do not justify a distributed container orchestrator. Setting it up here would multiply the operational complexity by five in exchange for solving problems Tramontana does not have.

This lesson is going to say so with numbers in section 2, and even so you are going to do the whole of it. For three reasons that do hold up:

  1. It is the industry's dominant technology. You are going to run into it in your next job, in most systems administration job adverts and in practically any platform team.
  2. Understanding it makes you a better administrator even if you never use it. Kubernetes is a distillation of decades of good practice — health, progressive deployments, resource limits, configuration separated from code — and those ideas apply just the same with systemd.
  3. Knowing when NOT to use it is part of knowing how to use it. And that can only be argued having built it.

So you build it on srv-tramontana-test, with k3s, and you deploy Tramontana Bookings for real. At the end you come back to the question knowing exactly what you are talking about.

Contents

  1. Objective, prerequisites and a warning
  2. What problem Kubernetes solves and when NOT to use it
  3. Architecture: control plane and worker nodes
  4. Distributions: kubeadm, k3s, microk8s and the managed ones
  5. Installing k3s with a server and an agent
  6. kubectl and the kubeconfig
  7. The objects, in dependency order
  8. Complete manifests for Tramontana Bookings
  9. Rolling deployment and rollback
  10. Security: securityContext, NetworkPolicy, RBAC and PSS
  11. Diagnosis: reading events and the four common failures
  12. Automation with Ansible
  13. The real operational complexity

Objective, prerequisites and a warning

Objective. Bring up a working two-node cluster, deploy Tramontana Bookings with internal high availability, TLS exposure, configuration and secrets separated, health probes and resource limits; carry out a deployment with no interruption and a rollback; and know how to diagnose the common failures.

Prerequisites: containers and Docker (07-05), especially namespaces, cgroups, capabilities and the multi-stage Dockerfile; the virtualisation environment from 07-04; the reverse proxy from 08-01; and the notion of health probes from 07-07.

An operational warning, and it is meant seriously: all of this is done on srv-tramontana-test. Kubernetes is not installed on srv-tramontana. k3s modifies the iptables/nftables rules, manages its own network interfaces and starts a container runtime of its own; on a production server with ufw, nftables, AppArmor and a service running, the interference is real and hard to undo.

$ ssh [email protected] 'hostnamectl; free -m | head -2; nproc'
 Static hostname: srv-tramontana-test
  Operating System: Ubuntu 24.04.1 LTS
            Kernel: Linux 6.8.0-41-generic
               total        used        free
Mem:            3891         412        3102
2

And a second VM for the worker node, created with cloud-init as in 07-04:

$ virt-install --name k8s-node2 --memory 2048 --vcpus 2 --disk size=20 \
    --cloud-init user-data=cloud-init-node2.yaml \
    --os-variant ubuntu24.04 --import --noautoconsole
$ virsh domifaddr k8s-node2
 vnet2  52:54:00:a1:b2:c4  ipv4  192.168.122.105/24

What problem Kubernetes solves and when NOT to use it

The real problem

Kubernetes was born to manage many containers on many machines declaratively. The problems it solves are concrete:

Problem Without Kubernetes With Kubernetes
Which machine do I start this container on? A person decides The scheduler, according to resources
A container dies Somebody restarts it It restarts by itself
A machine dies Somebody relocates its workloads They relocate themselves
Deploying without cutting the service A draining script (07-07) Native: RollingUpdate
Where is service X? A fixed IP or manual DNS Automatic internal DNS
Scaling from 3 to 30 replicas Provision and configure kubectl scale, seconds
Configuration and secrets Files per machine ConfigMap and Secret, versioned
The system's desired state Documentation and trust The cluster reconciles by itself

That last row is the central idea. Kubernetes is a reconciliation loop: you declare "I want three replicas of this image with these limits", and a set of controllers continuously compare the real state with the declared one and act to bring them closer together. You do not give orders; you describe an objective. It is the same idempotence principle as Ansible in 07-06, but continuous instead of one-off.

When NOT to use it, and Tramontana's case

Signal Kubernetes?
1-3 services, one or two machines No
A team with nobody dedicated to platform work No
A stable, predictable load No
A monolithic application with state on the local disk No, or with a great deal of work
Dozens of services and teams Yes
Frequent, unpredictable scaling Yes
Deployments several times a day Yes
Multi-tenant with isolation Yes
You already use a provider with managed Kubernetes Probably yes

The analysis for Tramontana, with numbers:

Item Current situation With Kubernetes
Services to orchestrate 1 application + 1 database The same
Machines needed 1 (2 with the replica) 3 minimum for a control plane with quorum
New pieces to maintain — etcd, CNI, Ingress, internal certificates, CRDs
Upgrades per year apt upgrade 3-4 minor versions, each with breaking-change notes
Training needed You already have it 3-6 months until productive
Recovery time after a failure 50 min (measured, 07-06) It depends on whether the failure is the cluster's
Benefit in availability ~99.8 % with option B from 07-07 The same, with more complexity

The conclusion is unambiguous: Kubernetes would give Tramontana nothing that option B from 07-07 does not already give — two application nodes with a load balancer — and it would add an entire control plane to maintain. The professional recommendation is still the one from 07-07.

And the corollary that is worth internalising: Kubernetes is not the natural evolution of a well-administered system. It is a tool for a specific problem of scale. A well-maintained three-machine system with systemd, Ansible and a load balancer is an excellent architecture, not an immature stage.

Architecture: control plane and worker nodes

graph TB
    subgraph CP["CONTROL PLANE (server node)"]
        API["kube-apiserver<br/>the only way in<br/>validates, authenticates, persists"]
        ETCD[("etcd<br/>key-value database<br/>ALL the state lives here")]
        SCH["kube-scheduler<br/>decides WHICH NODE<br/>each Pod goes to"]
        CM["kube-controller-manager<br/>reconciliation loops<br/>real vs. desired"]
        API <--> ETCD
        SCH --> API
        CM --> API
    end
    subgraph N1["WORKER NODE 1"]
        K1["kubelet<br/>agent: starts and watches<br/>this node's Pods"]
        P1["kube-proxy<br/>network rules<br/>for the Services"]
        R1["containerd<br/>container runtime"]
        K1 --> R1
    end
    subgraph N2["WORKER NODE 2"]
        K2["kubelet"]
        P2["kube-proxy"]
        R2["containerd"]
        K2 --> R2
    end
    K1 -.->|"'what am I<br/>supposed to run?'"| API
    K2 -.-> API
    P1 -.-> API
    P2 -.-> API
    USER["kubectl apply -f app.yaml"] --> API

The control plane, the four pieces you have to know:

Component What it does If it fails
kube-apiserver The only way in. It authenticates, authorises, validates and writes to etcd. Everything goes through here Nothing can be changed; what is already running carries on running
etcd A key-value database with all the cluster's state The cluster is unrecoverable without a backup
kube-scheduler Chooses a node for each new Pod according to resources, affinities and constraints New Pods stay in Pending
kube-controller-manager Loops that reconcile the real and desired states (replicas, nodes, endpoints) Nothing self-heals

The worker nodes:

Component What it does
kubelet The agent on each node: it asks the apiserver what it has to run, starts containers, runs the probes and reports the state
kube-proxy Programs the network rules (iptables or IPVS) that make the Services work
runtime (containerd) Runs the containers. It is what you saw in 07-05, without Docker in the middle

Two observations that put the mental model in order:

Everything goes through the apiserver. There is no direct communication between components: the scheduler does not talk to the kubelet, it writes to the apiserver that that Pod goes to that node, and the kubelet reads it. That central-bus architecture is what makes the system extensible and auditable.

If the control plane goes down, the workload carries on working. The kubelets carry on running what they already had and restarting whatever falls over. What is lost is the ability to change things and to react to whole-node failures. It is a valuable design property and it surprises a lot of people.

Distributions: kubeadm, k3s, microk8s and the managed ones

kubeadm k3s microk8s minikube EKS/GKE/AKS
Who maintains it The project SUSE (Rancher) Canonical The project The provider
Installation Several steps One command A snap One command A web console
Binary size ~1 GB in pieces ~70 MB, a single one ~200 MB Variable —
Realistic minimum RAM 2 GB per node 512 MB 1 GB 2 GB —
State store etcd SQLite by default, etcd optional dqlite etcd Managed
Real multi-node Yes Yes Yes No (it is local) Yes
Conformance certificate Yes Yes Yes Yes Yes
Production Yes Yes, with caveats Yes No Yes
Ships Ingress and a load balancer No Yes: Traefik + ServiceLB With addons With addons Yes
Cost of the control plane The hardware The hardware The hardware — 70-100 €/month

The choice is k3s, for four concrete reasons:

  1. It fits on the test machine. With 3.8 GB and 2 vCPU, kubeadm is a tight squeeze; k3s leaves room to deploy something on top.
  2. One binary and one command, without ceasing to be certified Kubernetes: the manifests are identical and what you learn transfers unchanged.
  3. It ships the essentials already assembled: Traefik as the Ingress, ServiceLB for LoadBalancer-type Services, local-path as the StorageClass and CoreDNS. With kubeadm you would have to install and configure each piece, which is instructive and here is noise.
  4. It is real production, not a toy: it is used in edge deployments, in factories and on devices.

Its trade-off: SQLite instead of etcd by default, which means a single control-plane node. For learning and for many real cases, that is plenty; for high availability of the control plane you have to move to embedded etcd with three nodes.

About the managed services (EKS, GKE, AKS): the provider maintains the control plane — upgrades, certificates, etcd, backups — and you only supply worker nodes. It costs about 70-100 € a month per cluster, and for a small company that genuinely needs Kubernetes it is usually the sensible option: most of the operational complexity of section 13 stops being your problem.

Installing k3s with a server and an agent

# ===== SERVER NODE: srv-tramontana-test (192.168.122.104) =====
$ ssh [email protected]

# Download and REVIEW the script before running it. Piping a URL
# straight into sh is exactly what 05-03 advises against.
$ curl -sfL https://get.k3s.io -o /tmp/k3s-install.sh
$ sha256sum /tmp/k3s-install.sh
$ less /tmp/k3s-install.sh

$ INSTALL_K3S_VERSION="v1.30.4+k3s1" \
  INSTALL_K3S_EXEC="server \
      --write-kubeconfig-mode 0644 \
      --disable traefik=false \
      --node-label role=server \
      --tls-san 192.168.122.104" \
  sh /tmp/k3s-install.sh
[INFO]  Using v1.30.4+k3s1 as release
[INFO]  systemd: Starting k3s

$ sudo systemctl status k3s --no-pager | head -4
● k3s.service - Lightweight Kubernetes
     Active: active (running) since Tue 2026-08-18 15:02:11 CEST

Pinning the version with INSTALL_K3S_VERSION is not optional. Without it the latest one is installed, and a machine rebuilt six months later would get a different version: exactly the problem Ansible came to solve in 07-06. It is the same pinning policy as 05-03.

# The token that authenticates the nodes that join. It is a real secret:
# with it, anybody can join a node to the cluster.
$ sudo cat /var/lib/rancher/k3s/server/node-token
K10a3f19c8d::server:8f2b1c4d5e6a7b8c9d0e1f2a3b4c5d6e
# ===== AGENT NODE: k8s-node2 (192.168.122.105) =====
$ ssh [email protected]
$ curl -sfL https://get.k3s.io -o /tmp/k3s-install.sh
$ INSTALL_K3S_VERSION="v1.30.4+k3s1" \
  K3S_URL="https://192.168.122.104:6443" \
  K3S_TOKEN="K10a3f19c8d::server:8f2b1c4d5e6a7b8c9d0e1f2a3b4c5d6e" \
  INSTALL_K3S_EXEC="agent --node-label role=worker" \
  sh /tmp/k3s-install.sh
# ===== Verification, from the server =====
$ sudo k3s kubectl get nodes -o wide
NAME                  STATUS   ROLES                  AGE   VERSION        INTERNAL-IP
srv-tramontana-test   Ready    control-plane,master   4m    v1.30.4+k3s1   192.168.122.104
k8s-node2             Ready    <none>                 1m    v1.30.4+k3s1   192.168.122.105

$ sudo k3s kubectl get pods -A
NAMESPACE     NAME                                     READY   STATUS      RESTARTS
kube-system   coredns-6799fbcd5-x8k2p                  1/1     Running     0
kube-system   local-path-provisioner-6c86858495-mn4qt  1/1     Running     0
kube-system   metrics-server-54fd9b65b-7jc9d           1/1     Running     0
kube-system   svclb-traefik-4a1b2c3d-p9k4m             2/2     Running     0
kube-system   traefik-7d764994d8-lm2xw                 1/1     Running     0

Resources consumed by the empty cluster, which is an honest figure worth having:

$ sudo k3s kubectl top nodes
NAME                  CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
srv-tramontana-test   142m         7%     812Mi           21%
k8s-node2             38m          1%     284Mi           14%

812 MiB of RAM and 7 % of CPU without deploying anything. With kubeadm it would be 1.5-2 GB. That is the entry price of orchestration, and it is part of the argument in section 2.

kubectl and the kubeconfig

# From your laptop, without depending on sudo on the server
$ sudo apt install kubernetes-client   # or download the official binary
$ mkdir -p ~/.kube
$ scp [email protected]:/etc/rancher/k3s/k3s.yaml ~/.kube/config-test
$ sed -i 's/127.0.0.1/192.168.122.104/' ~/.kube/config-test
$ chmod 0600 ~/.kube/config-test
$ export KUBECONFIG=~/.kube/config-test

The kubeconfig is a YAML file with three lists — clusters, users and contexts — and an active context. It contains cluster administrator credentials, so it goes with 600 permissions and never into a repository:

# ~/.kube/config-test (fragment)
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBD...   # the cluster's CA
    server: https://192.168.122.104:6443
  name: default
users:
- name: default
  user:
    client-certificate-data: LS0tLS1CRUdJTiBD...      # your certificate
    client-key-data: LS0tLS1CRUdJTiBS...              # YOUR PRIVATE KEY
contexts:
- context: {cluster: default, user: default, namespace: tramontana}
  name: test
current-context: test

The commands used 95 % of the time:

Command What for
kubectl get <kind> List. -o wide adds columns; -o yaml gives the complete object
kubectl describe <kind> <name> Detail and events: the first command when there is a problem
kubectl logs <pod> Logs. -f follows; --previous shows those of the previous container
kubectl exec -it <pod> -- sh A shell inside the container
kubectl apply -f <file> Apply a manifest declaratively
kubectl delete -f <file> Remove what it declares
kubectl get events --sort-by=.lastTimestamp What has happened, in order
kubectl -n <namespace> Work in another namespace
$ kubectl get nodes
NAME                  STATUS   ROLES                  AGE   VERSION
srv-tramontana-test   Ready    control-plane,master   12m   v1.30.4+k3s1
k8s-node2             Ready    <none>                 9m    v1.30.4+k3s1

# Completion and an alias: it gets used dozens of times a day
$ echo 'source <(kubectl completion bash)' >> ~/.bashrc
$ echo 'alias k=kubectl; complete -o default -F __start_kubectl k' >> ~/.bashrc

--previous deserves a note: when a container restarts in a loop, kubectl logs shows those of the current attempt, which is usually empty because it has just started. The ones that explain the failure are those of the previous attempt, and they can only be seen with --previous. It is the first genuinely useful diagnostic trick in Kubernetes.

The objects, in dependency order

Pod: the unit, and why it is almost never created by hand

A Pod is one or more containers that share network space (the same IP, the same localhost), storage and life cycle. It is the smallest unit Kubernetes schedules.

$ kubectl run test --image=nginx:1.27-alpine --restart=Never
$ kubectl get pod test -o wide
NAME   READY   STATUS    RESTARTS   AGE   IP           NODE
test   1/1     Running   0          8s    10.42.1.14   k8s-node2

$ kubectl delete pod test
$ kubectl get pods
No resources found.

It has gone and it does not come back. That is the reason not to create Pods directly: a Pod is ephemeral and nobody watches it. If the node dies, the Pod dies with it. What you create is a higher-level object that guarantees that N Pods exist, whatever happens.

ReplicaSet and Deployment

A ReplicaSet maintains N identical replicas. It is not created by hand either: it is managed by a Deployment, which on top of that knows how to do rolling deployments and rollbacks.

Deployment  →  manages ReplicaSets (one per version)  →  they create Pods

When the image changes, the Deployment creates a new ReplicaSet and reduces the old one while it increases the new one. The old ones are kept with zero replicas, which is what makes kubectl rollout undo possible.

Strategy Behaviour When
RollingUpdate (the default) Replaces gradually. Both versions coexist The usual case
Recreate Kills everything and starts the new one. There is an outage When two versions cannot coexist (an incompatible schema migration)
strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 1          # how many EXTRA Pods there may be temporarily
    maxUnavailable: 0    # how many FEWER. 0 = full capacity at all times

maxUnavailable: 0 with maxSurge: 1 is the no-interruption deployment configuration: first a new one starts, and only when it is ready is an old one withdrawn. It is exactly the connection draining from 07-07, automated.

Service and the internal DNS

Pods have IPs and they change constantly. A Service gives a stable name and a virtual IP to a set of Pods, selected by labels.

Type Reach Use
ClusterIP (the default) Only inside the cluster Communication between services
NodePort A high port (30000-32767) on every node Testing; not very elegant
LoadBalancer Requests an external load balancer In the cloud; in k3s, ServiceLB provides it
ExternalName A DNS alias to an external name Referencing the database outside the cluster

How the internal DNS works, which is one of the most elegant things about Kubernetes: CoreDNS automatically creates a record per Service with the pattern

<service>.<namespace>.svc.cluster.local
$ kubectl run dns-test --rm -it --image=busybox:1.36 --restart=Never -- \
      nslookup tramontana.tramontana.svc.cluster.local
Server:    10.43.0.10
Address:   10.43.0.10:53
Name:      tramontana.tramontana.svc.cluster.local
Address:   10.43.112.88

From a Pod in the same namespace, tramontana is enough; from another one, tramontana.tramontana. An IP is never hard-coded: you use the name, and the cluster takes care of it.

And what happens underneath: kube-proxy programs rules on each node that rewrite the destination towards one of the Pod IPs behind the Service. There is no load-balancing process; it is the kernel itself.

Ingress and its relationship with the Nginx from 08-01

A LoadBalancer-type Service per application means one public IP per application. An Ingress acts as a shared HTTP reverse proxy: it routes by domain name and by path towards different Services, and it terminates TLS.

An Ingress is only a declaration; a controller is needed to implement it. k3s ships Traefik.

And here is the connection with 08-01, which is the natural question: an Ingress does exactly what you did with Nginx — terminate TLS, route by domain, add headers, rate-limit — with two differences:

The Nginx from 08-01 Kubernetes Ingress
Configuration A file in /etc/nginx/ A declarative object in the cluster
Destinations Fixed IP and port in an upstream Services, which follow the Pods
On deployment nginx -s reload Nothing: the endpoints update themselves
Certificates certbot on the disk A Secret, or cert-manager automatically
Who applies it systemd The controller (Traefik, ingress-nginx)

In fact, the most widely used controller is ingress-nginx, which is Nginx underneath generating its configuration from cluster objects. What you learned in 08-01 applies literally; only who writes the file changes.

ConfigMap and Secret, and the serious warning

ConfigMap Secret
What for Non-sensitive configuration Passwords, keys, certificates
Storage Plain text in etcd base64 in etcd
Maximum size 1 MiB 1 MiB
Can be mounted as Environment variables or files The same

A Kubernetes Secret is base64, NOT encryption. Base64 is an encoding, not cryptography: it is undone with one command and without any key.

$ kubectl create secret generic test --from-literal=password='SuperSecret123'
$ kubectl get secret test -o jsonpath='{.data.password}' | base64 -d; echo
SuperSecret123

There it is, in the clear, with one command and no additional credential. The practical consequences, linking directly back to 06-05:

  • Never push a Secret to Git. A YAML file with data: in base64 is a file with the password in it.
  • By default it is stored unencrypted in etcd. Whoever reads the etcd file or its backup has every secret in the cluster.
  • RBAC is what really protects them: only somebody with read permission on Secrets sees them.

The three real solutions, in increasing order of rigour:

Solution How it works Assessment
Encryption at rest for etcd EncryptionConfiguration in the apiserver The indispensable minimum in production
Sealed Secrets Encrypted with the cluster's public key; the encrypted YAML can go to Git Very practical
An external store (Vault, the provider's secrets) The Pods request them at startup; they are never in etcd The most robust

It is the same conclusion as 06-05 in a different wrapper: secrets do not live next to the code, and pass with systemd-creds solved exactly this problem outside Kubernetes.

PersistentVolumeClaim, StorageClass and Namespace

A container is ephemeral: what it writes to its file system disappears when it restarts. A PersistentVolumeClaim (PVC) is a request for persistent storage; a StorageClass defines how it is provisioned.

$ kubectl get storageclass
NAME                   PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE
local-path (default)   rancher.io/local-path   Delete          WaitForFirstConsumer

A warning about local-path: it creates a directory on the node's disk. It is fast and it has one serious limitation: the Pod becomes tied to that node. If the node dies, the data is nowhere else. For genuinely distributed storage you need Longhorn, Ceph or the cloud provider's storage.

A Namespace is a logical division of the cluster: it separates names, allows resource quotas and is the natural unit of RBAC and NetworkPolicy.

Complete manifests for Tramontana Bookings

# ~/tramontana-k8s/00-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: tramontana
  labels:
    # Pod Security Standards (section 10): rejects privileged Pods
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
---
# Quota: stops a configuration mistake consuming the whole cluster
apiVersion: v1
kind: ResourceQuota
metadata:
  name: tramontana-quota
  namespace: tramontana
spec:
  hard:
    requests.cpu: "2"
    requests.memory: 2Gi
    limits.cpu: "4"
    limits.memory: 4Gi
    persistentvolumeclaims: "4"
# ~/tramontana-k8s/10-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: tramontana-config
  namespace: tramontana
data:
  # It is the same old /etc/tramontana/app.conf, as a cluster object
  db_host: "10.0.2.15"
  db_port: "6432"              # PgBouncer (08-02)
  db_name: "tramontana"
  max_connections: "80"
  query_timeout: "30"
  log_level: "info"
  listen: "0.0.0.0"            # inside the Pod, not the host
  port: "8080"
---
apiVersion: v1
kind: Secret
metadata:
  name: tramontana-secrets
  namespace: tramontana
type: Opaque
stringData:
  # stringData lets you write in the clear and Kubernetes encodes it.
  # THIS FILE DOES NOT GO TO GIT. In production, Sealed Secrets or Vault.
  db_password: "PLACEHOLDER_INJECTED_FROM_PASS"
# In practice, the Secret is created with no intermediate file:
$ kubectl -n tramontana create secret generic tramontana-secrets \
      --from-literal=db_password="$(pass tramontana/db)" \
      --dry-run=client -o yaml | kubectl apply -f -

That --dry-run=client -o yaml | kubectl apply -f - is the idiomatic pattern for creating or updating a Secret without it ending up in the history or on disk.

# ~/tramontana-k8s/20-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tramontana
  namespace: tramontana
  labels:
    app: tramontana
spec:
  replicas: 2
  revisionHistoryLimit: 5        # how many old ReplicaSets to keep
  selector:
    matchLabels:
      app: tramontana

  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1                # one extra during the deployment
      maxUnavailable: 0          # NEVER less capacity than declared

  template:
    metadata:
      labels:
        app: tramontana
        version: "3.2.1"
    spec:
      # Spread the replicas across different nodes: if one node goes
      # down, both of them do not go with it.
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: tramontana

      # --- Pod-level security (section 10) ---
      securityContext:
        runAsNonRoot: true
        runAsUser: 997             # svc-tramontana, the usual uid
        runAsGroup: 1002           # the tramontana group
        fsGroup: 1002
        seccompProfile:
          type: RuntimeDefault

      containers:
        - name: tramontana
          image: registry.tramontana.example/tramontana:3.2.1
          # NEVER ':latest'. A moving tag makes two Pods of the same
          # "version" run different code, and it makes a rollback roll
          # back to nowhere.
          imagePullPolicy: IfNotPresent

          ports:
            - name: http
              containerPort: 8080

          # --- Configuration from the ConfigMap ---
          envFrom:
            - configMapRef:
                name: tramontana-config
          env:
            - name: TRAMONTANA_DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: tramontana-secrets
                  key: db_password

          # --- Resources: the relationship with the cgroups of 07-05 ---
          # requests: what the SCHEDULER reserves in order to pick a node.
          # limits:   the real ceiling, imposed by cgroups v2.
          #   - Exceeding the MEMORY limit -> the kernel KILLS the
          #     container (OOMKilled): it is a hard limit.
          #   - Exceeding the CPU one -> it is THROTTLED, not killed:
          #     the process simply runs more slowly.
          resources:
            requests:
              cpu: 100m            # 0.1 of a core
              memory: 256Mi
            limits:
              cpu: 500m
              memory: 512Mi

          # --- The three probes, each answering something different ---

          # startupProbe: "has it finished starting?" While it fails,
          # the other two do NOT run. It is what allows slow starts
          # without relaxing liveness. 30 x 5 s = 150 s of headroom.
          startupProbe:
            httpGet: {path: /health, port: http}
            periodSeconds: 5
            failureThreshold: 30

          # readinessProbe: "can it serve requests NOW?"
          # If it fails, the Pod LEAVES the Service (it stops receiving
          # traffic) but it is NOT restarted. It is the balancer's gate.
          readinessProbe:
            httpGet: {path: /health, port: http}
            periodSeconds: 5
            timeoutSeconds: 2
            successThreshold: 1
            failureThreshold: 3

          # livenessProbe: "is it alive or hung?" If it fails, the
          # kubelet KILLS AND RESTARTS the container. That is why it is
          # the most dangerous: badly calibrated, it causes cascading
          # restarts under load. LOOSER thresholds than readiness,
          # always.
          livenessProbe:
            httpGet: {path: /health, port: http}
            periodSeconds: 15
            timeoutSeconds: 3
            failureThreshold: 5

          # --- Container-level security ---
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]

          # With readOnlyRootFilesystem you have to give explicit
          # writable space for whatever genuinely needs it.
          volumeMounts:
            - name: tmp
              mountPath: /tmp
            - name: cache
              mountPath: /var/cache/tramontana

          # An orderly shutdown: it gives time to finish the requests in
          # flight before the SIGTERM arrives.
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 5"]

      terminationGracePeriodSeconds: 30
      volumes:
        - name: tmp
          emptyDir: {sizeLimit: 64Mi}
        - name: cache
          emptyDir: {sizeLimit: 256Mi}
# ~/tramontana-k8s/30-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: tramontana
  namespace: tramontana
spec:
  type: ClusterIP
  selector:
    app: tramontana            # the Pod's labels, not the Deployment's
  ports:
    - name: http
      port: 80                 # the Service's port
      targetPort: http         # the container's named port
---
# The database lives OUTSIDE the cluster (srv-tramontana). This object
# gives it an internal name, so that the application always uses
# "postgresql" and not a hard-coded IP.
apiVersion: v1
kind: Service
metadata:
  name: postgresql
  namespace: tramontana
spec:
  type: ExternalName
  externalName: srv-tramontana.internal
# ~/tramontana-k8s/40-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tramontana
  namespace: tramontana
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.tls: "true"
    # The security headers from 08-01, as annotations
    traefik.ingress.kubernetes.io/router.middlewares: tramontana-security@kubernetescrd
spec:
  ingressClassName: traefik
  tls:
    - hosts: [bookings.tramontana.example]
      secretName: tramontana-tls      # a kubernetes.io/tls type Secret
  rules:
    - host: bookings.tramontana.example
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: tramontana
                port: {name: http}
---
# A Traefik Middleware: the equivalent of the add_header lines of 08-01
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: security
  namespace: tramontana
spec:
  headers:
    stsSeconds: 63072000
    stsIncludeSubdomains: true
    contentTypeNosniff: true
    frameDeny: true
    referrerPolicy: "strict-origin-when-cross-origin"
# The certificate from 06-05, as a Secret
$ sudo kubectl -n tramontana create secret tls tramontana-tls \
      --cert=/etc/letsencrypt/live/bookings.tramontana.example/fullchain.pem \
      --key=/etc/letsencrypt/live/bookings.tramontana.example/privkey.pem

# Deploy everything, in order
$ kubectl apply -f ~/tramontana-k8s/
namespace/tramontana created
resourcequota/tramontana-quota created
configmap/tramontana-config created
deployment.apps/tramontana created
service/tramontana created
ingress.networking.k8s.io/tramontana created

$ kubectl -n tramontana get pods -o wide
NAME                          READY   STATUS    RESTARTS   AGE   NODE
tramontana-7d8f9c4b5-k2m4p    1/1     Running   0          48s   srv-tramontana-test
tramontana-7d8f9c4b5-x9n7q    1/1     Running   0          48s   k8s-node2

$ kubectl -n tramontana get endpoints tramontana
NAME         ENDPOINTS                           AGE
tramontana   10.42.0.18:8080,10.42.1.22:8080     51s

The replicas have landed on different nodes thanks to topologySpreadConstraints, and the Service already has its two endpoints.

The three probes deserve a summary table, because confusing them is the most expensive mistake in this section:

Probe Question If it fails Risk if it is wrong
startupProbe Has it finished starting? Restarts once the headroom runs out Without it, liveness kills slow starts
readinessProbe Can it serve now? Leaves the Service, no restart Without it, traffic reaches a Pod that is not ready
livenessProbe Is it hung? Kills and restarts Too aggressive: cascading restarts under load

And the classic mistake, which turns up in real production systems with serious consequences: putting the same failureThreshold and periodSeconds on readiness and liveness. Under load the application answers a little more slowly, both fail, and the kubelet restarts Pods that were only busy — which reduces capacity, increases the load on the remaining ones and causes a cascading outage. Liveness always looser than readiness.

Rolling deployment and rollback

# 1. Deploy version 3.3.0. --record is deprecated; an annotation is
#    used instead so that the reason is on the record.
$ kubectl -n tramontana set image deployment/tramontana \
      tramontana=registry.tramontana.example/tramontana:3.3.0
$ kubectl -n tramontana annotate deployment/tramontana \
      kubernetes.io/change-cause="Version 3.3.0: optimised reports"

# 2. Follow it live
$ kubectl -n tramontana rollout status deployment/tramontana
Waiting for deployment "tramontana" rollout to finish: 1 out of 2 new replicas have been updated...
Waiting for deployment "tramontana" rollout to finish: 1 old replicas are pending termination...
deployment "tramontana" successfully rolled out

# 3. During the process both ReplicaSets coexist
$ kubectl -n tramontana get rs
NAME                    DESIRED   CURRENT   READY   AGE
tramontana-7d8f9c4b5    0         0         0       18m    # 3.2.1
tramontana-9f2a1c8e7    2         2         2       42s    # 3.3.0

Zero requests lost, and the mechanism is the one you already know from 07-07: maxUnavailable: 0 guarantees that there is never less capacity than declared, and the readinessProbe guarantees that a Pod receives no traffic until it answers. The draining you did in 07-07 with socat against HAProxy's socket is, here, a property of the system.

# 4. The new version fails in production: roll back
$ kubectl -n tramontana rollout history deployment/tramontana
REVISION  CHANGE-CAUSE
1         Initial version 3.2.1
2         Version 3.3.0: optimised reports

$ kubectl -n tramontana rollout undo deployment/tramontana
deployment.apps/tramontana rolled back

$ kubectl -n tramontana rollout status deployment/tramontana
deployment "tramontana" successfully rolled out

The rollback took eleven seconds, against the minutes of the deploy.sh from 04-07 with its ln -sfn. It is the strongest argument in favour of Kubernetes: reversibility is a property of the system, not a script you have to remember to write correctly.

With the caveat that always has to be stated: rollout undo reverts the code, not the data. If version 3.3.0 ran a schema migration, going back to 3.2.1 leaves the old application against a new database. It is the same problem as in 04-07, and the solution is the same: backwards-compatible migrations. Kubernetes does not solve it.

# 5. Scaling, which is trivial and that is why it catches the eye
$ kubectl -n tramontana scale deployment/tramontana --replicas=4
$ kubectl -n tramontana get pods --no-headers | wc -l
4

Security: securityContext, NetworkPolicy, RBAC and PSS

securityContext

Kubernetes' defaults are permissive: without a securityContext, a container runs as root with a broad set of capabilities. Everything from 07-05 applies here declaratively:

Directive What it prevents Equivalent in 07-05
runAsNonRoot: true The container running as root USER in the Dockerfile
runAsUser: 997 — --user
allowPrivilegeEscalation: false Gaining privileges via SUID --security-opt no-new-privileges
readOnlyRootFilesystem: true Writing to the file system --read-only
capabilities.drop: ["ALL"] Every kernel capability --cap-drop=ALL
seccompProfile: RuntimeDefault Dangerous system calls --security-opt seccomp

readOnlyRootFilesystem: true is the one that frustrates the most attacks in practice: most exploits need to write something to disk in order to persist or to download the next stage.

# Check that it really is applied
$ kubectl -n tramontana exec deploy/tramontana -- id
uid=997 gid=1002 groups=1002
$ kubectl -n tramontana exec deploy/tramontana -- touch /test
touch: /test: Read-only file system
command terminated with exit code 1

NetworkPolicy

By default, every Pod in the cluster can talk to every other one. It is a flat network, and in a cluster with several applications that means a compromised Pod reaches any other.

# ~/tramontana-k8s/50-networkpolicy.yaml
# 1. Deny EVERYTHING by default in the namespace (allowlist, 06-03)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all
  namespace: tramontana
spec:
  podSelector: {}                # every Pod in the namespace
  policyTypes: [Ingress, Egress]
---
# 2. Allow only what is needed
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-tramontana
  namespace: tramontana
spec:
  podSelector:
    matchLabels: {app: tramontana}
  policyTypes: [Ingress, Egress]
  ingress:
    # Only from the Ingress (Traefik lives in kube-system)
    - from:
        - namespaceSelector:
            matchLabels: {kubernetes.io/metadata.name: kube-system}
      ports: [{protocol: TCP, port: 8080}]
  egress:
    # DNS: without this, NOTHING works and the failure is baffling
    - to:
        - namespaceSelector:
            matchLabels: {kubernetes.io/metadata.name: kube-system}
      ports:
        - {protocol: UDP, port: 53}
        - {protocol: TCP, port: 53}
    # PostgreSQL, outside the cluster
    - to:
        - ipBlock: {cidr: 10.0.2.15/32}
      ports: [{protocol: TCP, port: 6432}]

Forgetting the DNS rule is the number one mistake with NetworkPolicy. The policy is applied, everything stops working, and the symptom — "the name cannot be resolved" — does not point at the firewall. With policyTypes: [Egress], traffic to CoreDNS is blocked too.

An important warning for k3s: the default CNI (Flannel) does not implement NetworkPolicy, so these objects are accepted and do nothing. You have to install k3s with --flannel-backend=none and deploy Calico, or use --disable-network-policy=false depending on the version. Checking it is essential: believing you have segmentation when you do not is worse than not having it.

RBAC, briefly

Role-based access control uses four objects:

Object Scope What it defines
Role One namespace Which verbs on which resources
ClusterRole The whole cluster The same, globally
RoleBinding One namespace Who has that Role
ClusterRoleBinding The whole cluster Who has that ClusterRole
# Luis: read-only and logs in his namespace. He cannot see Secrets.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: tramontana
  name: reader
rules:
  - apiGroups: ["", "apps"]
    resources: ["pods", "pods/log", "deployments", "services", "configmaps"]
    verbs: ["get", "list", "watch"]
  # 'secrets' does NOT appear: reading them is reading the passwords (base64)
# Check permissions, your own or somebody else's
$ kubectl auth can-i get secrets -n tramontana --as=luis
no
$ kubectl auth can-i get pods -n tramontana --as=luis
yes

kubectl auth can-i is the RBAC verification tool, and it is the way to check that what is forbidden is forbidden, just as you did with the PostgreSQL roles in 08-02.

Pod Security Standards

They replace PodSecurityPolicy, withdrawn in 1.25. They are three profiles applied per namespace with labels:

Profile What it allows
privileged Everything. No restrictions
baseline Blocks the most dangerous things: privileged, hostNetwork, hostPath
restricted Requires runAsNonRoot, drop ALL, seccomp, no escalation

The namespace's enforce: restricted makes the apiserver reject any Pod that does not comply. It is a safety net that does not depend on remembering to add the securityContext:

$ kubectl -n tramontana run bad --image=nginx --privileged
Error from server (Forbidden): pods "bad" is forbidden: violates PodSecurity
"restricted:latest": privileged (container "bad" must not set
securityContext.privileged=true), allowPrivilegeEscalation != false,
unrestricted capabilities, runAsNonRoot != true, seccompProfile

Diagnosis: reading events and the four common failures

The first command for any problem is kubectl describe, and what matters is at the bottom, in Events.

$ kubectl -n tramontana describe pod tramontana-7d8f9c4b5-k2m4p | tail -12
Events:
  Type     Reason     Age                From     Message
  ----     ------     ----               ----     -------
  Normal   Scheduled  2m                 default-scheduler  Successfully assigned...
  Normal   Pulling    2m                 kubelet  Pulling image "...:3.2.1"
  Normal   Pulled     1m                 kubelet  Successfully pulled image
  Normal   Created    1m                 kubelet  Created container tramontana
  Normal   Started    1m                 kubelet  Started container tramontana
  Warning  Unhealthy  30s (x3 over 40s)  kubelet  Readiness probe failed:
             Get "http://10.42.0.18:8080/health": dial tcp: connect: connection refused

That last line explains the whole problem: the application is not listening on 8080 yet.

The four failures you will genuinely see

1. CrashLoopBackOff — the container starts, dies, and Kubernetes waits longer and longer before retrying (10 s, 20 s, 40 s... up to 5 min).

$ kubectl -n tramontana get pods
NAME                         READY   STATUS             RESTARTS      AGE
tramontana-9f2a1c8e7-p4k9m   0/1     CrashLoopBackOff   5 (48s ago)   4m

# The key: --previous. Without it you see the logs of the CURRENT
# attempt, which has just started and is empty.
$ kubectl -n tramontana logs tramontana-9f2a1c8e7-p4k9m --previous
FATAL: could not connect to the database: password authentication
failed for user "svc_tramontana"

# Confirm the cause
$ kubectl -n tramontana get secret tramontana-secrets -o jsonpath='{.data.db_password}' \
      | base64 -d | head -c8; echo '...'
PLACEHO...

There it is: the Secret has the manifest's placeholder, not the real password.

Cause of CrashLoopBackOff How it is confirmed
A configuration or credentials error logs --previous
A missing dependency (an unreachable DB) logs --previous + exec into another Pod
The container's command finishes describe: Exit Code: 0
A livenessProbe that is too aggressive describe: Unhealthy events before each restart
OOMKilled describe: Reason: OOMKilled

2. ImagePullBackOff / ErrImagePull

$ kubectl -n tramontana describe pod tramontana-x | grep -A3 Events
  Warning  Failed  30s  kubelet  Failed to pull image
    "registry.tramontana.example/tramontana:3.3.1": failed to resolve reference:
    unexpected status: 401 Unauthorized
Message Cause Solution
401 Unauthorized A missing registry credential imagePullSecrets
not found / manifest unknown The tag does not exist (a typo) Check with crane ls or skopeo
no such host The node's DNS does not resolve the registry Check on the node, not in the Pod
connection refused The registry is down, or a firewall Check from the node

3. Pending — the Pod exists and no node accepts it.

$ kubectl -n tramontana describe pod tramontana-y | tail -4
  Warning  FailedScheduling  20s  default-scheduler
    0/2 nodes are available: 1 Insufficient cpu, 1 Insufficient memory.
    preemption: 0/2 nodes are available: 2 No preemption victims found.

# How much is actually committed on each node
$ kubectl describe node k8s-node2 | grep -A6 'Allocated resources'
Allocated resources:
  Resource   Requests      Limits
  cpu        1750m (87%)   3200m (160%)
  memory     1894Mi (94%)  2560Mi (128%)

The scheduler reserves according to requests, not to actual usage. A node with 20 % of its CPU in use can reject Pods if its requests add up to 95 %. It is the most common confusion with Pending, and it is why setting generous requests "just in case" wastes the whole cluster.

Other causes of Pending: a PVC that cannot be provisioned, a nodeSelector that matches no node, or taints without the corresponding toleration.

4. OOMKilled — the kernel killed the container for exceeding the memory limit.

$ kubectl -n tramontana describe pod tramontana-z | grep -A4 'Last State'
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137            # 128 + 9 (SIGKILL)

It is the same cgroups mechanism as 07-05. Exit Code: 137 is the unmistakable signature. And the decision is not automatic: you have to work out whether the limit is too low or whether there is a memory leak in the application — raising the limit without looking only delays the problem.

The diagnostic toolbox

# The namespace's events, in chronological order
$ kubectl -n tramontana get events --sort-by=.lastTimestamp | tail -10

# Real consumption (it needs metrics-server, which k3s ships)
$ kubectl -n tramontana top pods
NAME                         CPU(cores)   MEMORY(bytes)
tramontana-7d8f9c4b5-k2m4p   12m          184Mi

# Debug the network from inside the cluster, without touching the real Pods
$ kubectl -n tramontana run debug --rm -it --image=nicolaka/netshoot \
      --restart=Never -- bash
debug:~# nslookup tramontana
debug:~# curl -sv http://tramontana/health
debug:~# nc -zv 10.0.2.15 6432

# Forward a port to your laptop, without exposing anything
$ kubectl -n tramontana port-forward svc/tramontana 8080:80
$ curl -s localhost:8080/health

port-forward is the most used day-to-day tool: it lets you reach an internal service from your laptop without creating an Ingress or a NodePort.

Automation with Ansible

There is an obvious temptation here and it is worth avoiding: Ansible must not replace kubectl apply. The manifests are already declarative and idempotent; wrapping them in Ansible adds a layer without gaining anything. The correct division:

Layer Tool Why
Preparing the node's operating system Ansible It is machine configuration
Installing and configuring k3s Ansible It is a systemd service
Joining nodes to the cluster Ansible It needs the token, which is a secret
Deploying applications kubectl apply or Helm It is already declarative
Keeping the cluster in sync with Git Argo CD or Flux (GitOps) It is the industry pattern
# ~/tramontana-infra/roles/k3s/defaults/main.yml
---
k3s_version: "v1.30.4+k3s1"
k3s_url: "https://192.168.122.104:6443"
k3s_datastore: sqlite
k3s_disable: []
k3s_extra_args_server: "--write-kubeconfig-mode 0644 --tls-san {{ ansible_default_ipv4.address }}"
# ~/tramontana-infra/roles/k3s/tasks/main.yml
---
- name: Safety guard — NEVER in production
  ansible.builtin.assert:
    that: inventory_hostname != 'srv-tramontana'
    fail_msg: >-
      This role must NOT be run on srv-tramontana. k3s rewrites the
      nftables rules and starts its own container runtime; on the
      production server it would interfere with ufw, AppArmor and the
      tramontana service.

- name: System requirements
  ansible.builtin.apt:
    name: [curl, iptables, apparmor-utils]
    state: present

- name: Download the k3s installer
  ansible.builtin.get_url:
    url: https://get.k3s.io
    dest: /usr/local/src/k3s-install.sh
    mode: '0755'

- name: Install k3s on the SERVER node
  ansible.builtin.command:
    cmd: /usr/local/src/k3s-install.sh
    creates: /usr/local/bin/k3s        # idempotence (07-06)
  environment:
    INSTALL_K3S_VERSION: "{{ k3s_version }}"
    INSTALL_K3S_EXEC: "server {{ k3s_extra_args_server }}"
  when: "'k3s_server' in group_names"

- name: Read the token from the server node
  ansible.builtin.slurp:
    src: /var/lib/rancher/k3s/server/node-token
  register: k3s_token_raw
  when: "'k3s_server' in group_names"

- name: Share the token with the agents
  ansible.builtin.set_fact:
    k3s_token: "{{ hostvars[groups['k3s_server'][0]]['k3s_token_raw']['content'] | b64decode | trim }}"
  no_log: true

- name: Install k3s on the AGENT nodes
  ansible.builtin.command:
    cmd: /usr/local/src/k3s-install.sh
    creates: /usr/local/bin/k3s
  environment:
    INSTALL_K3S_VERSION: "{{ k3s_version }}"
    K3S_URL: "{{ k3s_url }}"
    K3S_TOKEN: "{{ k3s_token }}"
    INSTALL_K3S_EXEC: "agent"
  no_log: true
  when: "'k3s_agent' in group_names"

- name: Wait for the node to be Ready
  ansible.builtin.command: "k3s kubectl get node {{ ansible_hostname }} -o json"
  register: node
  until: >-
    (node.stdout | from_json).status.conditions
    | selectattr('type','eq','Ready') | map(attribute='status') | first == 'True'
  retries: 30
  delay: 10
  changed_when: false
  delegate_to: "{{ groups['k3s_server'][0] }}"

# --- A backup of the cluster, which is what really matters ---
- name: Install the etcd/SQLite backup script
  ansible.builtin.copy:
    src: k3s_backup.sh
    dest: /usr/local/bin/k3s_backup.sh
    mode: '0750'
  when: "'k3s_server' in group_names"

- name: Daily timer for backing up the cluster state
  ansible.builtin.copy:
    src: "{{ item }}"
    dest: "/etc/systemd/system/{{ item }}"
    mode: '0644'
  loop: [k3s-backup.service, k3s-backup.timer]
  notify: Reload systemd
  when: "'k3s_server' in group_names"

That initial assert is the literal application of the warning in section 1, turned into code that cannot be ignored by carelessness. And backing up the cluster state is not optional: without etcd or its SQLite, the cluster does not exist.

# roles/k3s/files/k3s_backup.sh (the core)
$ k3s etcd-snapshot save --name daily           # with etcd
# or, with SQLite:
$ sqlite3 /var/lib/rancher/k3s/server/db/state.db ".backup '/srv/backups/k3s.db'"
$ restic backup /srv/backups/k3s.db --tag k3s

The real operational complexity

Here is the part Kubernetes presentations leave out, and the one that really answers the question in section 2.

1. The upgrades are constant. Kubernetes releases three minor versions a year and each one gets around fourteen months of support. That forces you to upgrade the cluster at least twice a year, indefinitely. Each upgrade means reading the release notes looking for breaking changes, upgrading the control plane and then the nodes, and verifying that the workloads still work.

2. API versions get withdrawn, and they break manifests that used to work.

Resource Old API Current API Withdrawn in
Ingress extensions/v1beta1 networking.k8s.io/v1 1.22
CronJob batch/v1beta1 batch/v1 1.25
PodSecurityPolicy policy/v1beta1 Removed: use PSS 1.25
HorizontalPodAutoscaler autoscaling/v2beta2 autoscaling/v2 1.26

A manifest written two years ago can fail after an upgrade. There are tools (pluto, kubent) to detect it beforehand, and using them is mandatory.

3. etcd is delicate. It is a distributed database with quorum: it needs 3 or 5 nodes (an odd number), it is sensitive to disk latency, and it requires periodic compaction and defragmentation. A corrupted etcd with no backup is a lost cluster. And everything about quorum from 07-07 applies.

4. The internal certificates expire. Kubernetes uses mutual TLS between all of its components, with a PKI of its own and certificates that expire after a year. kubeadm renews them on upgrade, but a cluster that has not been touched for fourteen months stops working all at once, with baffling authentication errors.

$ sudo kubeadm certs check-expiration | head -5
CERTIFICATE                EXPIRES                  RESIDUAL TIME
apiserver                  Aug 18, 2027 13:02 UTC   364d
etcd-server                Aug 18, 2027 13:02 UTC   364d

5. The ecosystem moves very fast. Docker Shim withdrawn in 1.24, PSP replaced by PSS, Ingress evolving towards the Gateway API. What you learn goes out of date faster than in any other area of the system.

6. Diagnosing demands understanding more layers. A failure can be in the application, the container, the Pod, the Service, the CNI, the Ingress, the DNS, the scheduler or the kubelet. Everything from Module 7 is still necessary; Kubernetes adds layers, it does not replace them.

The estimated annual cost, in hours, which is the figure you have to take into a decision:

Activity Hours/year
Two minor version upgrades 16-32
Maintaining etcd and its backups 8-12
Updating manifests for withdrawn APIs 4-8
Diagnosing cluster-specific incidents 20-40
Continuous training 20-40
Total 70-130 h/year

Between two and four weeks of work a year, just to keep the cluster alive, without counting deploying applications. With a managed service, half that. With systemd and Ansible on three machines, practically zero.

That number is the final answer to the question in section 2, and it also explains why managed Kubernetes is so successful: for most companies, 100 € a month is cheaper than 100 hours a year.

Common Mistakes and Tips

  • Using :latest for the image. Two Pods of the same "version" can run different code, and rollout undo rolls back to nowhere. Immutable tags, always.
  • Creating loose Pods instead of Deployments. Nobody watches them: if they die, they do not come back.
  • A livenessProbe equal to or stricter than the readinessProbe. Under load, healthy Pods get restarted and the service falls over in a cascade. Liveness always looser.
  • Not putting a startupProbe on slow-starting applications. Liveness kills the container before it finishes starting, in a loop.
  • Believing a Secret is encrypted. It is base64. kubectl get secret -o jsonpath | base64 -d and there it is.
  • Pushing Secrets to Git. The YAML with data: is the file with the password. Sealed Secrets or an external store.
  • Forgetting the DNS rule in an Egress NetworkPolicy. Everything stops working and the symptom does not point at the firewall.
  • Applying NetworkPolicy with a CNI that does not implement it. Flannel in k3s accepts them and ignores them: you think you have segmentation and you do not.
  • requests equal to limits and very generous. The scheduler reserves by requests: you waste the cluster and cause Pending with empty nodes.
  • Raising the memory limit on seeing an OOMKilled without investigating. It could be a leak, and you are only delaying the problem.
  • kubectl logs without --previous on a CrashLoopBackOff. You see the logs of the current attempt, which is empty.
  • Not backing up etcd or the SQLite. Without it, the cluster cannot be rebuilt.
  • Ignoring the expiry of the internal certificates. A cluster untouched for fourteen months stops working all at once.
  • Installing Kubernetes on an existing production server. It rewrites network rules and starts its own runtime. A dedicated machine.
  • Setting up Kubernetes for three services. It is the architectural mistake of this entire lesson: 70-130 h/year to solve problems you do not have.
  • A tip on method. Faced with any failure: kubectl describe and read the Events from the bottom up. It solves most problems before you touch anything.

Exercises

Exercise 1

After deploying version 3.3.0, the Pods go into CrashLoopBackOff and the service goes down. Diagnose the problem methodically, restore the service and explain what would have prevented the incident.

Exercise 2

Compare, for Tramontana's case, the Kubernetes architecture from this lesson with option B recommended in 07-07, using objective criteria, and issue a recommendation.

Exercise 3

Write a CronJob manifest that runs the database backup inside the cluster, with the same guarantees as tramontana-backup.timer from 05-05, and explain what is gained and what is lost compared with the systemd timer.

Solutions

Solution 1

Step 1: the overall picture, before touching anything.

$ kubectl -n tramontana get pods
NAME                          READY   STATUS             RESTARTS      AGE
tramontana-7d8f9c4b5-k2m4p    1/1     Running            0             3h    # 3.2.1
tramontana-9f2a1c8e7-p4k9m    0/1     CrashLoopBackOff   4 (52s ago)   3m    # 3.3.0

First important and reassuring fact: thanks to maxUnavailable: 0, the version 3.2.1 Pod is still running and serving traffic. The deployment is blocked, not down. That takes the pressure off and allows you to diagnose without rushing.

$ kubectl -n tramontana rollout status deployment/tramontana --timeout=10s
error: timed out waiting for the condition

$ kubectl -n tramontana get endpoints tramontana
NAME         ENDPOINTS          AGE
tramontana   10.42.0.18:8080    3h

A single endpoint: the new Pod has never reached Ready, so the Service has never sent it traffic. The readinessProbe has done exactly its job.

Step 2: the logs of the previous attempt.

$ kubectl -n tramontana logs tramontana-9f2a1c8e7-p4k9m --previous
[2026-08-18T16:02:11Z] INFO  Tramontana Bookings 3.3.0 starting
[2026-08-18T16:02:11Z] INFO  reading configuration from the environment
[2026-08-18T16:02:11Z] ERROR variable TRAMONTANA_CACHE_URL not defined
[2026-08-18T16:02:11Z] FATAL incomplete configuration; aborting

The root cause in four lines. Version 3.3.0 introduces a new configuration variable — TRAMONTANA_CACHE_URL, for the report cache — that is not in the ConfigMap. The application fails fast and clearly, which is the correct behaviour.

Step 3: confirm and rule out other causes.

$ kubectl -n tramontana describe pod tramontana-9f2a1c8e7-p4k9m | \
      grep -A6 'Last State'
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1              # NOT 137: it is not OOMKilled
      Started:      Tue, 18 Aug 2026 16:02:11 +0200
      Finished:     Tue, 18 Aug 2026 16:02:11 +0200

$ kubectl -n tramontana get configmap tramontana-config -o jsonpath='{.data}' | \
      jq 'keys'
["db_host","db_name","db_port","listen","log_level","max_connections",
 "port","query_timeout"]

Exit Code: 1 with the start and the stop in the same second rules out OOMKilled (137), badly calibrated probes (it would have reached Running for a while) and network problems (it never got as far as connecting to anything). And the ConfigMap confirms the missing key.

Step 4: decide. And the correct decision is to roll back first.

$ kubectl -n tramontana rollout undo deployment/tramontana
deployment.apps/tramontana rolled back
$ kubectl -n tramontana rollout status deployment/tramontana
deployment "tramontana" successfully rolled out

Why roll back before fixing, even though the fix looks trivial: the Deployment is in an inconsistent state, with a new ReplicaSet trying to start in a loop every few seconds. Each retry generates events, consumes resources and pollutes the diagnosis. Leaving the system in a known, stable state before correcting it is the discipline that stops you making things worse — and it is the same logic as the decision tree in the recovery runbook from 07-06.

Step 5: the fix, tested before applying it.

$ kubectl -n tramontana patch configmap tramontana-config \
      --type merge -p '{"data":{"cache_url":"redis://cache.tramontana.svc:6379"}}'
# 20-deployment.yaml — the explicit mapping of the variable
          env:
            - name: TRAMONTANA_CACHE_URL
              valueFrom:
                configMapKeyRef:
                  name: tramontana-config
                  key: cache_url
# Verify in a throwaway Pod BEFORE touching the Deployment
$ kubectl -n tramontana run verify --rm -it --restart=Never \
      --image=registry.tramontana.example/tramontana:3.3.0 \
      --overrides='{"spec":{"containers":[{"name":"verify",
        "image":"registry.tramontana.example/tramontana:3.3.0",
        "envFrom":[{"configMapRef":{"name":"tramontana-config"}}],
        "env":[{"name":"TRAMONTANA_CACHE_URL","valueFrom":
          {"configMapKeyRef":{"name":"tramontana-config","key":"cache_url"}}}]}]}}' \
      -- /usr/bin/tramontana --check-config
configuration valid
# Now yes
$ kubectl apply -f ~/tramontana-k8s/20-deployment.yaml
$ kubectl -n tramontana rollout status deployment/tramontana
deployment "tramontana" successfully rolled out
$ kubectl -n tramontana get pods
NAME                          READY   STATUS    RESTARTS   AGE
tramontana-3c7e9a2b1-h4j8k    1/1     Running   0          42s
tramontana-3c7e9a2b1-w2n5r    1/1     Running   0          28s

One detail worth knowing: changing a ConfigMap does not restart the Pods that use it as environment variables. If that had been needed:

$ kubectl -n tramontana rollout restart deployment/tramontana

What would have prevented the incident, which is the question that really matters:

Measure How it would have prevented it
Testing the manifest in a test namespace The failure would have happened where it does not matter
Validating the configuration when the image starts A --check-config in the build fails before deploying
Release notes with the configuration changes Luis must document every new variable
A default value in the application A new variable with a reasonable value breaks nothing
CI that applies the manifests to an ephemeral cluster The failure is caught in the code review
An alert on the first CrashLoopBackOff It does not prevent it, but it detects it in a minute

And the three lessons of method:

  1. maxUnavailable: 0 turned an outage into a blocked deployment. It is the line of configuration with the best benefit/cost ratio in the whole manifest, and it deserves to be in any production Deployment.
  2. The readinessProbe stopped the Service sending traffic to a broken Pod. Without it, half the requests would have failed for three minutes.
  3. --previous was what gave the answer. Without that option, kubectl logs returned a freshly started, empty container, and the diagnosis would have taken far longer.

Solution 2

A comparison with objective criteria, on Tramontana's real case: a monolithic application, one database, ~500 bookings a month, two technical people.

Criterion Kubernetes (k3s) Option B from 07-07 (2 nodes + HAProxy)
Machines needed 3 (control plane with quorum) or 2 with a SPOF 3 (2 app + 1 load balancer)
Base RAM consumed 812 MiB for the cluster alone ~120 MiB (HAProxy + keepalived)
Deployment without interruption Native, RollingUpdate Manual draining with socat, a script
Rollback rollout undo, 11 s ln -sfn + restart, ~2 min
Self-healing of a Pod Automatic systemd's Restart=on-failure
Self-healing when a node goes down Automatic, it relocates HAProxy takes it out; it does not relocate
Scaling to 4 replicas One command, seconds Provision a machine + Ansible, ~1 h
Learning curve 3-6 months You already have it
Annual maintenance 70-130 h ~15 h
New pieces to master etcd, CNI, Ingress, RBAC, PSS, CRDs None
Diagnosing a failure More layers to rule out The usual ones
Measured full rebuild Unmeasured; the cluster adds steps 50 min, measured
Achievable availability ~99.8-99.9 % ~99.8 %
Estimated annual cost 3 VMs + 100 h of work 3 VMs + 15 h
Transferable to another job A great deal Moderate

The four points where Kubernetes genuinely wins, without exaggerating:

  1. Deployment and rollback. Eleven seconds against two minutes, and with no script of your own to maintain. It is a real difference.
  2. Relocation when a node goes down. HAProxy takes the failed node out of the rotation but does not recover the capacity; Kubernetes starts the missing replicas on the surviving node.
  3. Scaling. One command against an hour of work. It only matters if scaling is frequent, and at Tramontana it is not.
  4. Employability. It is a legitimate personal argument but it is not an architectural argument for the company, and they have to be separated honestly.

The five where it loses:

  1. 812 MiB of RAM before deploying anything, on 3.8 GB machines. That is more than 20 % of the server devoted to orchestration.
  2. 70-130 hours a year of indefinite maintenance, against about 15.
  3. Three to six months before you can resolve incidents fluently. During that period, reliability goes down.
  4. More failure modes. It is the same argument as 07-07: a system with more pieces has more ways of breaking, and Kubernetes adds a lot of them.
  5. The database is still outside. The hard problem from 07-07 — state — is not solved by Kubernetes. PostgreSQL would still be on srv-tramontana, with its manual promotion and its RPO.

The analysis that settles the question, and it is the same reasoning as in 07-07: both options reach roughly the same availability, around 99.8 %, because most of Tramontana's interruptions do not come from a machine failing but from deployments and maintenance, and both of them solve those. Kubernetes does it more elegantly and automatically; option B does it with tools the team already masters.

When two options give the same result, the simpler one wins. And the difference in maintenance — between 55 and 115 hours a year — is equivalent to two or three weeks of work that would not be spent improving the product.

Recommendation: keep option B from 07-07 for production.

And keep the k3s cluster on srv-tramontana-test as a learning lab, at zero marginal cost, because the machine already exists. It is where to practise without risk and where to be ready for the scenario that would change the recommendation.

The three circumstances that would change it, written down in advance so that they can be recognised:

Change Why it would change the decision
Tramontana splits into 5+ services Orchestration starts to pay for itself
Managed Kubernetes is contracted in the cloud 60 % of the maintenance disappears
Somebody with real experience joins The learning curve stops being a cost

And a final warning worth putting in writing: today's correct decision may not be correct in two years, and that does not mean it is badly taken today. Setting an annual review is more professional than choosing the architecture you might need one day.

Solution 3

# ~/tramontana-k8s/60-cronjob-backup.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: database-backup
  namespace: tramontana
spec:
  # 02:30, the same as tramontana-backup.timer (05-05).
  # CAREFUL: the schedule is interpreted in the CLUSTER'S TIME ZONE,
  # which is UTC by default. timeZone (stable since 1.27) solves it;
  # without it, the backup would run at 04:30 in summer.
  schedule: "30 2 * * *"
  timeZone: "Europe/Madrid"

  # The equivalent of the flock from 04-07: if yesterday's backup is
  # still running, another one is NOT launched on top of it.
  concurrencyPolicy: Forbid

  # If the cluster was down at 02:30, does it run when it comes back?
  # 3600 = only if less than 1 h has passed. It is the equivalent of
  # the timer's Persistent=true, but BOUNDED: without this field, a
  # badly delayed Job would be launched at an inconvenient moment.
  startingDeadlineSeconds: 3600

  successfulJobsHistoryLimit: 7    # keep 7 successful runs
  failedJobsHistoryLimit: 14       # and 14 failed ones: those are the ones you look at

  jobTemplate:
    spec:
      # Retries on failure, with exponential backoff
      backoffLimit: 2
      # An absolute ceiling: a hung backup does not block the next one
      activeDeadlineSeconds: 3600
      # Automatic cleanup of the Job 3 days later
      ttlSecondsAfterFinished: 259200

      template:
        spec:
          restartPolicy: OnFailure

          securityContext:
            runAsNonRoot: true
            runAsUser: 997
            runAsGroup: 1002
            fsGroup: 1002
            seccompProfile: {type: RuntimeDefault}

          containers:
            - name: backup
              image: registry.tramontana.example/backup:1.4.0
              command: ["/usr/local/bin/backup_tramontana.sh"]

              env:
                - name: PGHOST
                  valueFrom:
                    configMapKeyRef: {name: tramontana-config, key: db_host}
                - name: PGDATABASE
                  valueFrom:
                    configMapKeyRef: {name: tramontana-config, key: db_name}
                - name: PGPASSWORD
                  valueFrom:
                    secretKeyRef: {name: tramontana-secrets, key: db_password}
                - name: RESTIC_REPOSITORY
                  valueFrom:
                    secretKeyRef: {name: restic-secrets, key: repository}
                - name: RESTIC_PASSWORD
                  valueFrom:
                    secretKeyRef: {name: restic-secrets, key: password}

              resources:
                requests: {cpu: 100m, memory: 256Mi}
                # The backup compresses: it needs more headroom than the app
                limits:   {cpu: "1",  memory: 1Gi}

              securityContext:
                allowPrivilegeEscalation: false
                readOnlyRootFilesystem: true
                capabilities: {drop: ["ALL"]}

              volumeMounts:
                - {name: work, mountPath: /tmp}
                - {name: restic-cache, mountPath: /var/cache/restic}

          volumes:
            - name: work
              emptyDir: {sizeLimit: 4Gi}
            # The restic cache speeds up successive backups a great
            # deal: persisting it between runs is a real optimisation.
            - name: restic-cache
              persistentVolumeClaim: {claimName: restic-cache}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: restic-cache
  namespace: tramontana
spec:
  accessModes: [ReadWriteOnce]
  resources: {requests: {storage: 2Gi}}
# Test it WITHOUT waiting until 02:30 (the equivalent of systemctl start)
$ kubectl -n tramontana create job --from=cronjob/database-backup manual-test
$ kubectl -n tramontana logs job/manual-test -f
[2026-08-18 17:12:03] starting the PostgreSQL backup
[2026-08-18 17:12:41] base backup verified: 1.7 GiB
[2026-08-18 17:14:12] restic: snapshot a3f19c8d saved
[2026-08-18 17:14:20] backup completed

$ kubectl -n tramontana get cronjob
NAME              SCHEDULE     TIMEZONE        SUSPEND   ACTIVE   LAST SCHEDULE
database-backup   30 2 * * *   Europe/Madrid   False     0        8h

What is gained and what is lost, which is the central part of the exercise:

Kubernetes CronJob tramontana-backup.timer (systemd)
Runs if a node goes down Yes, on another node No: it dies with the machine
Time zone timeZone (1.27+); UTC by default The system's zone, naturally
Mutual exclusion concurrencyPolicy: Forbid flock (04-07)
Delayed execution startingDeadlineSeconds Persistent=true
Run history Job objects you can query journalctl -u
Retries backoffLimit, native Restart= + RestartSec
Resource isolation requests/limits CPUQuota, MemoryMax
Secrets Secret in base64 systemd-creds, encrypted (06-05)
Failure notification Requires external monitoring OnFailure=, native
Access to the host's disk Complicated, and rightly so Direct
Debugging kubectl logs job/... journalctl, or run the script
Pieces needed Cluster + registry + image A script and a .timer file

What is genuinely gained is two things, and only two:

  1. Resistance to a node failure. It is the strong argument: tramontana-backup.timer lives on srv-tramontana and if that machine is switched off at 02:30, there is no backup that night and nobody finds out until check_backup.sh detects it at 08:00. The CronJob runs on any available node.
  2. Real isolation and limits, just as easily. Although systemd also gives them with MemoryMax, here they come as standard.

What is lost, and it carries weight:

  1. The secrets get worse. systemd-creds from 06-05 stores them encrypted with the TPM or a host key; a Kubernetes Secret is base64 in etcd. It is an objective step backwards, unless encryption at rest or Vault is added.
  2. Failure notification stops being native. OnFailure= in systemd sends the alert directly; with a CronJob you have to monitor kube_job_status_failed from Prometheus (08-06). One more piece.
  3. Reaching the host's disk gets complicated, and rightly so: mounting /srv/tramontana/backups in a Pod requires a hostPath, which the restricted Pod Security Standards forbid. The flow would have to be rewritten so that the backup goes straight to restic without passing through the local disk, which is cleaner but is work.
  4. An image has to be built and published with the script, pg_dump, restic and their dependencies — and kept up to date. The timer uses what is already installed.

A recommendation consistent with solution 2:

Keep tramontana-backup.timer in systemd. The only real benefit — resistance to a node failure — can be achieved without Kubernetes: by running the timer on the second application node too with a shared lock, or by launching it from a machine other than the one being backed up, which is also better design because the backup does not depend on the backed-up server being alive.

And there is one additional argument: the backup must carry on working when the cluster fails. A backup mechanism that depends on the infrastructure it backs up is exactly the kind of coupling that has to be avoided. It is the same principle by which the runbook from 05-08 lives outside the server and the AIDE database outside the machine.

Conclusion

You have built a two-node Kubernetes cluster, you have deployed Tramontana Bookings on top of it with complete manifests, you have carried out a rolling deployment without losing a single request and a rollback in eleven seconds. And you know what is underneath: the apiserver as the only way in, etcd holding all the state, the scheduler choosing a node according to requests, the controllers tirelessly reconciling the real with the declared, and the kubelet running on each node whatever it is given. You know that if the control plane goes down, the workload carries on working — and why.

You understand the objects in their dependency order and why each one exists. That nobody watches a loose Pod. That the Deployment manages ReplicaSets and that rollout undo lives there. That a Service gives a stable name and IP to a moving target, and that the internal DNS makes it transparent. That an Ingress is the Nginx from 08-01 written as a declarative object — in fact, the most widely used controller is Nginx underneath. And you know that a Secret is base64, not encryption, which links directly to pass and systemd-creds from 06-05 and explains why Sealed Secrets and Vault exist.

You have mastered the three probes and the difference that separates them, which is what causes the most incidents in real production systems: readiness takes you out of the rotation, liveness kills and restarts, startup gives the start-up headroom — and liveness always looser than readiness, or cascading restarts under load are only a matter of time. You know that requests is what the scheduler reserves and limits what the cgroup imposes, with the key asymmetry: exceeding the memory kills the container, exceeding the CPU only throttles it. And you know how to read a CrashLoopBackOff with --previous, a Pending by looking at the node's committed requests, and an Exit Code: 137 as the OOM killer's signature.

But the most valuable thing in this lesson is the conclusion it started with: for Tramontana, Kubernetes is disproportionate, and now you can defend that with numbers — 812 MiB of RAM before deploying anything, 70 to 130 hours of maintenance a year, three to six months of learning curve — instead of with an intuition. That the professional answer to "should we use Kubernetes?" can be "no, and here is why" is exactly the same criterion with which you answered Marta about high availability in 07-07. When two options give the same result, the simpler one wins. And the test cluster stays, because a lab where you can practise without risk is worth far more than the decision not to use it in production.

In 08-06 the course closes. You are going to take everything built across eight modules into a going-to-production checklist of some thirty rows, each with its verifiable evidence — and that is where the encryption-in-transit debt that 08-01 resolved is formally closed. You will finally set up the monitoring with Prometheus and Grafana that was postponed in 05-07, with the four golden signals, the essential PromQL and metrics of your own: the age of the last backup, the days until the certificate expires, the replica's lag. You will configure alerts that are actionable, which is the opposite of what almost everybody does. You will write the daily operations procedure, the duty log and the blameless post-incident review. You will turn the 99.8 % proposed in 07-07 into an error budget that decides whether you deploy or stabilise. And you will close the three debts still open since Module 5: the off-site backup that is not append-only, the authorized_keys2 that nobody has explained, and the availability objective that was never formalised.

Linux Course: From Beginner to System Administrator

Module 1: Introduction to Linux

Module 2: Basic Linux Commands

Module 3: Advanced Command-Line Skills

Module 4: Shell Scripting

Module 5: System Administration

Module 6: Networking and Security

Module 7: Advanced Topics

Module 8: Practical Projects

© Copyright 2026. All rights reserved