Let us begin with what no Kubernetes lesson usually says in its first paragraph: for Tramontana S.L., Kubernetes is disproportionate. One application, one database, three people and five hundred bookings a month do not justify a distributed container orchestrator. Setting it up here would multiply the operational complexity by five in exchange for solving problems Tramontana does not have.
This lesson is going to say so with numbers in section 2, and even so you are going to do the whole of it. For three reasons that do hold up:
- It is the industry's dominant technology. You are going to run into it in your next job, in most systems administration job adverts and in practically any platform team.
- Understanding it makes you a better administrator even if you never use it. Kubernetes is a distillation of decades of good practice — health, progressive deployments, resource limits, configuration separated from code — and those ideas apply just the same with systemd.
- Knowing when NOT to use it is part of knowing how to use it. And that can only be argued having built it.
So you build it on srv-tramontana-test, with k3s, and you deploy Tramontana Bookings for real. At the end you come back to the question knowing exactly what you are talking about.
Contents
- Objective, prerequisites and a warning
- What problem Kubernetes solves and when NOT to use it
- Architecture: control plane and worker nodes
- Distributions: kubeadm, k3s, microk8s and the managed ones
- Installing k3s with a server and an agent
- kubectl and the kubeconfig
- The objects, in dependency order
- Complete manifests for Tramontana Bookings
- Rolling deployment and rollback
- Security: securityContext, NetworkPolicy, RBAC and PSS
- Diagnosis: reading events and the four common failures
- Automation with Ansible
- The real operational complexity
Objective, prerequisites and a warning
Objective. Bring up a working two-node cluster, deploy Tramontana Bookings with internal high availability, TLS exposure, configuration and secrets separated, health probes and resource limits; carry out a deployment with no interruption and a rollback; and know how to diagnose the common failures.
Prerequisites: containers and Docker (07-05), especially namespaces, cgroups, capabilities and the multi-stage Dockerfile; the virtualisation environment from 07-04; the reverse proxy from 08-01; and the notion of health probes from 07-07.
An operational warning, and it is meant seriously: all of this is done on srv-tramontana-test. Kubernetes is not installed on srv-tramontana. k3s modifies the iptables/nftables rules, manages its own network interfaces and starts a container runtime of its own; on a production server with ufw, nftables, AppArmor and a service running, the interference is real and hard to undo.
$ ssh [email protected] 'hostnamectl; free -m | head -2; nproc'
Static hostname: srv-tramontana-test
Operating System: Ubuntu 24.04.1 LTS
Kernel: Linux 6.8.0-41-generic
total used free
Mem: 3891 412 3102
2And a second VM for the worker node, created with cloud-init as in 07-04:
$ virt-install --name k8s-node2 --memory 2048 --vcpus 2 --disk size=20 \
--cloud-init user-data=cloud-init-node2.yaml \
--os-variant ubuntu24.04 --import --noautoconsole
$ virsh domifaddr k8s-node2
vnet2 52:54:00:a1:b2:c4 ipv4 192.168.122.105/24What problem Kubernetes solves and when NOT to use it
The real problem
Kubernetes was born to manage many containers on many machines declaratively. The problems it solves are concrete:
| Problem | Without Kubernetes | With Kubernetes |
|---|---|---|
| Which machine do I start this container on? | A person decides | The scheduler, according to resources |
| A container dies | Somebody restarts it | It restarts by itself |
| A machine dies | Somebody relocates its workloads | They relocate themselves |
| Deploying without cutting the service | A draining script (07-07) | Native: RollingUpdate |
| Where is service X? | A fixed IP or manual DNS | Automatic internal DNS |
| Scaling from 3 to 30 replicas | Provision and configure | kubectl scale, seconds |
| Configuration and secrets | Files per machine | ConfigMap and Secret, versioned |
| The system's desired state | Documentation and trust | The cluster reconciles by itself |
That last row is the central idea. Kubernetes is a reconciliation loop: you declare "I want three replicas of this image with these limits", and a set of controllers continuously compare the real state with the declared one and act to bring them closer together. You do not give orders; you describe an objective. It is the same idempotence principle as Ansible in 07-06, but continuous instead of one-off.
When NOT to use it, and Tramontana's case
| Signal | Kubernetes? |
|---|---|
| 1-3 services, one or two machines | No |
| A team with nobody dedicated to platform work | No |
| A stable, predictable load | No |
| A monolithic application with state on the local disk | No, or with a great deal of work |
| Dozens of services and teams | Yes |
| Frequent, unpredictable scaling | Yes |
| Deployments several times a day | Yes |
| Multi-tenant with isolation | Yes |
| You already use a provider with managed Kubernetes | Probably yes |
The analysis for Tramontana, with numbers:
| Item | Current situation | With Kubernetes |
|---|---|---|
| Services to orchestrate | 1 application + 1 database | The same |
| Machines needed | 1 (2 with the replica) | 3 minimum for a control plane with quorum |
| New pieces to maintain | — | etcd, CNI, Ingress, internal certificates, CRDs |
| Upgrades per year | apt upgrade |
3-4 minor versions, each with breaking-change notes |
| Training needed | You already have it | 3-6 months until productive |
| Recovery time after a failure | 50 min (measured, 07-06) | It depends on whether the failure is the cluster's |
| Benefit in availability | ~99.8 % with option B from 07-07 | The same, with more complexity |
The conclusion is unambiguous: Kubernetes would give Tramontana nothing that option B from 07-07 does not already give — two application nodes with a load balancer — and it would add an entire control plane to maintain. The professional recommendation is still the one from 07-07.
And the corollary that is worth internalising: Kubernetes is not the natural evolution of a well-administered system. It is a tool for a specific problem of scale. A well-maintained three-machine system with systemd, Ansible and a load balancer is an excellent architecture, not an immature stage.
Architecture: control plane and worker nodes
graph TB
subgraph CP["CONTROL PLANE (server node)"]
API["kube-apiserver<br/>the only way in<br/>validates, authenticates, persists"]
ETCD[("etcd<br/>key-value database<br/>ALL the state lives here")]
SCH["kube-scheduler<br/>decides WHICH NODE<br/>each Pod goes to"]
CM["kube-controller-manager<br/>reconciliation loops<br/>real vs. desired"]
API <--> ETCD
SCH --> API
CM --> API
end
subgraph N1["WORKER NODE 1"]
K1["kubelet<br/>agent: starts and watches<br/>this node's Pods"]
P1["kube-proxy<br/>network rules<br/>for the Services"]
R1["containerd<br/>container runtime"]
K1 --> R1
end
subgraph N2["WORKER NODE 2"]
K2["kubelet"]
P2["kube-proxy"]
R2["containerd"]
K2 --> R2
end
K1 -.->|"'what am I<br/>supposed to run?'"| API
K2 -.-> API
P1 -.-> API
P2 -.-> API
USER["kubectl apply -f app.yaml"] --> API
The control plane, the four pieces you have to know:
| Component | What it does | If it fails |
|---|---|---|
| kube-apiserver | The only way in. It authenticates, authorises, validates and writes to etcd. Everything goes through here | Nothing can be changed; what is already running carries on running |
| etcd | A key-value database with all the cluster's state | The cluster is unrecoverable without a backup |
| kube-scheduler | Chooses a node for each new Pod according to resources, affinities and constraints | New Pods stay in Pending |
| kube-controller-manager | Loops that reconcile the real and desired states (replicas, nodes, endpoints) | Nothing self-heals |
The worker nodes:
| Component | What it does |
|---|---|
| kubelet | The agent on each node: it asks the apiserver what it has to run, starts containers, runs the probes and reports the state |
| kube-proxy | Programs the network rules (iptables or IPVS) that make the Services work |
| runtime (containerd) | Runs the containers. It is what you saw in 07-05, without Docker in the middle |
Two observations that put the mental model in order:
Everything goes through the apiserver. There is no direct communication between components: the scheduler does not talk to the kubelet, it writes to the apiserver that that Pod goes to that node, and the kubelet reads it. That central-bus architecture is what makes the system extensible and auditable.
If the control plane goes down, the workload carries on working. The kubelets carry on running what they already had and restarting whatever falls over. What is lost is the ability to change things and to react to whole-node failures. It is a valuable design property and it surprises a lot of people.
Distributions: kubeadm, k3s, microk8s and the managed ones
kubeadm |
k3s | microk8s | minikube | EKS/GKE/AKS | |
|---|---|---|---|---|---|
| Who maintains it | The project | SUSE (Rancher) | Canonical | The project | The provider |
| Installation | Several steps | One command | A snap | One command | A web console |
| Binary size | ~1 GB in pieces | ~70 MB, a single one | ~200 MB | Variable | — |
| Realistic minimum RAM | 2 GB per node | 512 MB | 1 GB | 2 GB | — |
| State store | etcd | SQLite by default, etcd optional | dqlite | etcd | Managed |
| Real multi-node | Yes | Yes | Yes | No (it is local) | Yes |
| Conformance certificate | Yes | Yes | Yes | Yes | Yes |
| Production | Yes | Yes, with caveats | Yes | No | Yes |
| Ships Ingress and a load balancer | No | Yes: Traefik + ServiceLB | With addons | With addons | Yes |
| Cost of the control plane | The hardware | The hardware | The hardware | — | 70-100 €/month |
The choice is k3s, for four concrete reasons:
- It fits on the test machine. With 3.8 GB and 2 vCPU,
kubeadmis a tight squeeze; k3s leaves room to deploy something on top. - One binary and one command, without ceasing to be certified Kubernetes: the manifests are identical and what you learn transfers unchanged.
- It ships the essentials already assembled: Traefik as the Ingress, ServiceLB for LoadBalancer-type Services,
local-pathas the StorageClass and CoreDNS. Withkubeadmyou would have to install and configure each piece, which is instructive and here is noise. - It is real production, not a toy: it is used in edge deployments, in factories and on devices.
Its trade-off: SQLite instead of etcd by default, which means a single control-plane node. For learning and for many real cases, that is plenty; for high availability of the control plane you have to move to embedded etcd with three nodes.
About the managed services (EKS, GKE, AKS): the provider maintains the control plane — upgrades, certificates, etcd, backups — and you only supply worker nodes. It costs about 70-100 € a month per cluster, and for a small company that genuinely needs Kubernetes it is usually the sensible option: most of the operational complexity of section 13 stops being your problem.
Installing k3s with a server and an agent
# ===== SERVER NODE: srv-tramontana-test (192.168.122.104) =====
$ ssh [email protected]
# Download and REVIEW the script before running it. Piping a URL
# straight into sh is exactly what 05-03 advises against.
$ curl -sfL https://get.k3s.io -o /tmp/k3s-install.sh
$ sha256sum /tmp/k3s-install.sh
$ less /tmp/k3s-install.sh
$ INSTALL_K3S_VERSION="v1.30.4+k3s1" \
INSTALL_K3S_EXEC="server \
--write-kubeconfig-mode 0644 \
--disable traefik=false \
--node-label role=server \
--tls-san 192.168.122.104" \
sh /tmp/k3s-install.sh
[INFO] Using v1.30.4+k3s1 as release
[INFO] systemd: Starting k3s
$ sudo systemctl status k3s --no-pager | head -4
● k3s.service - Lightweight Kubernetes
Active: active (running) since Tue 2026-08-18 15:02:11 CESTPinning the version with INSTALL_K3S_VERSION is not optional. Without it the latest one is installed, and a machine rebuilt six months later would get a different version: exactly the problem Ansible came to solve in 07-06. It is the same pinning policy as 05-03.
# The token that authenticates the nodes that join. It is a real secret:
# with it, anybody can join a node to the cluster.
$ sudo cat /var/lib/rancher/k3s/server/node-token
K10a3f19c8d::server:8f2b1c4d5e6a7b8c9d0e1f2a3b4c5d6e# ===== AGENT NODE: k8s-node2 (192.168.122.105) =====
$ ssh [email protected]
$ curl -sfL https://get.k3s.io -o /tmp/k3s-install.sh
$ INSTALL_K3S_VERSION="v1.30.4+k3s1" \
K3S_URL="https://192.168.122.104:6443" \
K3S_TOKEN="K10a3f19c8d::server:8f2b1c4d5e6a7b8c9d0e1f2a3b4c5d6e" \
INSTALL_K3S_EXEC="agent --node-label role=worker" \
sh /tmp/k3s-install.sh# ===== Verification, from the server =====
$ sudo k3s kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP
srv-tramontana-test Ready control-plane,master 4m v1.30.4+k3s1 192.168.122.104
k8s-node2 Ready <none> 1m v1.30.4+k3s1 192.168.122.105
$ sudo k3s kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS
kube-system coredns-6799fbcd5-x8k2p 1/1 Running 0
kube-system local-path-provisioner-6c86858495-mn4qt 1/1 Running 0
kube-system metrics-server-54fd9b65b-7jc9d 1/1 Running 0
kube-system svclb-traefik-4a1b2c3d-p9k4m 2/2 Running 0
kube-system traefik-7d764994d8-lm2xw 1/1 Running 0Resources consumed by the empty cluster, which is an honest figure worth having:
$ sudo k3s kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
srv-tramontana-test 142m 7% 812Mi 21%
k8s-node2 38m 1% 284Mi 14%812 MiB of RAM and 7 % of CPU without deploying anything. With kubeadm it would be 1.5-2 GB. That is the entry price of orchestration, and it is part of the argument in section 2.
kubectl and the kubeconfig
# From your laptop, without depending on sudo on the server
$ sudo apt install kubernetes-client # or download the official binary
$ mkdir -p ~/.kube
$ scp [email protected]:/etc/rancher/k3s/k3s.yaml ~/.kube/config-test
$ sed -i 's/127.0.0.1/192.168.122.104/' ~/.kube/config-test
$ chmod 0600 ~/.kube/config-test
$ export KUBECONFIG=~/.kube/config-testThe kubeconfig is a YAML file with three lists — clusters, users and contexts — and an active context. It contains cluster administrator credentials, so it goes with 600 permissions and never into a repository:
# ~/.kube/config-test (fragment)
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBD... # the cluster's CA
server: https://192.168.122.104:6443
name: default
users:
- name: default
user:
client-certificate-data: LS0tLS1CRUdJTiBD... # your certificate
client-key-data: LS0tLS1CRUdJTiBS... # YOUR PRIVATE KEY
contexts:
- context: {cluster: default, user: default, namespace: tramontana}
name: test
current-context: testThe commands used 95 % of the time:
| Command | What for |
|---|---|
kubectl get <kind> |
List. -o wide adds columns; -o yaml gives the complete object |
kubectl describe <kind> <name> |
Detail and events: the first command when there is a problem |
kubectl logs <pod> |
Logs. -f follows; --previous shows those of the previous container |
kubectl exec -it <pod> -- sh |
A shell inside the container |
kubectl apply -f <file> |
Apply a manifest declaratively |
kubectl delete -f <file> |
Remove what it declares |
kubectl get events --sort-by=.lastTimestamp |
What has happened, in order |
kubectl -n <namespace> |
Work in another namespace |
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
srv-tramontana-test Ready control-plane,master 12m v1.30.4+k3s1
k8s-node2 Ready <none> 9m v1.30.4+k3s1
# Completion and an alias: it gets used dozens of times a day
$ echo 'source <(kubectl completion bash)' >> ~/.bashrc
$ echo 'alias k=kubectl; complete -o default -F __start_kubectl k' >> ~/.bashrc--previous deserves a note: when a container restarts in a loop, kubectl logs shows those of the current attempt, which is usually empty because it has just started. The ones that explain the failure are those of the previous attempt, and they can only be seen with --previous. It is the first genuinely useful diagnostic trick in Kubernetes.
The objects, in dependency order
Pod: the unit, and why it is almost never created by hand
A Pod is one or more containers that share network space (the same IP, the same localhost), storage and life cycle. It is the smallest unit Kubernetes schedules.
$ kubectl run test --image=nginx:1.27-alpine --restart=Never
$ kubectl get pod test -o wide
NAME READY STATUS RESTARTS AGE IP NODE
test 1/1 Running 0 8s 10.42.1.14 k8s-node2
$ kubectl delete pod test
$ kubectl get pods
No resources found.It has gone and it does not come back. That is the reason not to create Pods directly: a Pod is ephemeral and nobody watches it. If the node dies, the Pod dies with it. What you create is a higher-level object that guarantees that N Pods exist, whatever happens.
ReplicaSet and Deployment
A ReplicaSet maintains N identical replicas. It is not created by hand either: it is managed by a Deployment, which on top of that knows how to do rolling deployments and rollbacks.
When the image changes, the Deployment creates a new ReplicaSet and reduces the old one while it increases the new one. The old ones are kept with zero replicas, which is what makes kubectl rollout undo possible.
| Strategy | Behaviour | When |
|---|---|---|
RollingUpdate (the default) |
Replaces gradually. Both versions coexist | The usual case |
Recreate |
Kills everything and starts the new one. There is an outage | When two versions cannot coexist (an incompatible schema migration) |
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1 # how many EXTRA Pods there may be temporarily
maxUnavailable: 0 # how many FEWER. 0 = full capacity at all timesmaxUnavailable: 0 with maxSurge: 1 is the no-interruption deployment configuration: first a new one starts, and only when it is ready is an old one withdrawn. It is exactly the connection draining from 07-07, automated.
Service and the internal DNS
Pods have IPs and they change constantly. A Service gives a stable name and a virtual IP to a set of Pods, selected by labels.
| Type | Reach | Use |
|---|---|---|
| ClusterIP (the default) | Only inside the cluster | Communication between services |
| NodePort | A high port (30000-32767) on every node | Testing; not very elegant |
| LoadBalancer | Requests an external load balancer | In the cloud; in k3s, ServiceLB provides it |
| ExternalName | A DNS alias to an external name | Referencing the database outside the cluster |
How the internal DNS works, which is one of the most elegant things about Kubernetes: CoreDNS automatically creates a record per Service with the pattern
$ kubectl run dns-test --rm -it --image=busybox:1.36 --restart=Never -- \
nslookup tramontana.tramontana.svc.cluster.local
Server: 10.43.0.10
Address: 10.43.0.10:53
Name: tramontana.tramontana.svc.cluster.local
Address: 10.43.112.88From a Pod in the same namespace, tramontana is enough; from another one, tramontana.tramontana. An IP is never hard-coded: you use the name, and the cluster takes care of it.
And what happens underneath: kube-proxy programs rules on each node that rewrite the destination towards one of the Pod IPs behind the Service. There is no load-balancing process; it is the kernel itself.
Ingress and its relationship with the Nginx from 08-01
A LoadBalancer-type Service per application means one public IP per application. An Ingress acts as a shared HTTP reverse proxy: it routes by domain name and by path towards different Services, and it terminates TLS.
An Ingress is only a declaration; a controller is needed to implement it. k3s ships Traefik.
And here is the connection with 08-01, which is the natural question: an Ingress does exactly what you did with Nginx — terminate TLS, route by domain, add headers, rate-limit — with two differences:
| The Nginx from 08-01 | Kubernetes Ingress | |
|---|---|---|
| Configuration | A file in /etc/nginx/ |
A declarative object in the cluster |
| Destinations | Fixed IP and port in an upstream |
Services, which follow the Pods |
| On deployment | nginx -s reload |
Nothing: the endpoints update themselves |
| Certificates | certbot on the disk | A Secret, or cert-manager automatically |
| Who applies it | systemd | The controller (Traefik, ingress-nginx) |
In fact, the most widely used controller is ingress-nginx, which is Nginx underneath generating its configuration from cluster objects. What you learned in 08-01 applies literally; only who writes the file changes.
ConfigMap and Secret, and the serious warning
| ConfigMap | Secret | |
|---|---|---|
| What for | Non-sensitive configuration | Passwords, keys, certificates |
| Storage | Plain text in etcd | base64 in etcd |
| Maximum size | 1 MiB | 1 MiB |
| Can be mounted as | Environment variables or files | The same |
A Kubernetes Secret is base64, NOT encryption. Base64 is an encoding, not cryptography: it is undone with one command and without any key.
$ kubectl create secret generic test --from-literal=password='SuperSecret123'
$ kubectl get secret test -o jsonpath='{.data.password}' | base64 -d; echo
SuperSecret123There it is, in the clear, with one command and no additional credential. The practical consequences, linking directly back to 06-05:
- Never push a Secret to Git. A YAML file with
data:in base64 is a file with the password in it. - By default it is stored unencrypted in etcd. Whoever reads the etcd file or its backup has every secret in the cluster.
- RBAC is what really protects them: only somebody with read permission on Secrets sees them.
The three real solutions, in increasing order of rigour:
| Solution | How it works | Assessment |
|---|---|---|
| Encryption at rest for etcd | EncryptionConfiguration in the apiserver |
The indispensable minimum in production |
| Sealed Secrets | Encrypted with the cluster's public key; the encrypted YAML can go to Git | Very practical |
| An external store (Vault, the provider's secrets) | The Pods request them at startup; they are never in etcd | The most robust |
It is the same conclusion as 06-05 in a different wrapper: secrets do not live next to the code, and pass with systemd-creds solved exactly this problem outside Kubernetes.
PersistentVolumeClaim, StorageClass and Namespace
A container is ephemeral: what it writes to its file system disappears when it restarts. A PersistentVolumeClaim (PVC) is a request for persistent storage; a StorageClass defines how it is provisioned.
$ kubectl get storageclass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE
local-path (default) rancher.io/local-path Delete WaitForFirstConsumerA warning about local-path: it creates a directory on the node's disk. It is fast and it has one serious limitation: the Pod becomes tied to that node. If the node dies, the data is nowhere else. For genuinely distributed storage you need Longhorn, Ceph or the cloud provider's storage.
A Namespace is a logical division of the cluster: it separates names, allows resource quotas and is the natural unit of RBAC and NetworkPolicy.
Complete manifests for Tramontana Bookings
# ~/tramontana-k8s/00-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: tramontana
labels:
# Pod Security Standards (section 10): rejects privileged Pods
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/enforce-version: latest
---
# Quota: stops a configuration mistake consuming the whole cluster
apiVersion: v1
kind: ResourceQuota
metadata:
name: tramontana-quota
namespace: tramontana
spec:
hard:
requests.cpu: "2"
requests.memory: 2Gi
limits.cpu: "4"
limits.memory: 4Gi
persistentvolumeclaims: "4"# ~/tramontana-k8s/10-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: tramontana-config
namespace: tramontana
data:
# It is the same old /etc/tramontana/app.conf, as a cluster object
db_host: "10.0.2.15"
db_port: "6432" # PgBouncer (08-02)
db_name: "tramontana"
max_connections: "80"
query_timeout: "30"
log_level: "info"
listen: "0.0.0.0" # inside the Pod, not the host
port: "8080"
---
apiVersion: v1
kind: Secret
metadata:
name: tramontana-secrets
namespace: tramontana
type: Opaque
stringData:
# stringData lets you write in the clear and Kubernetes encodes it.
# THIS FILE DOES NOT GO TO GIT. In production, Sealed Secrets or Vault.
db_password: "PLACEHOLDER_INJECTED_FROM_PASS"# In practice, the Secret is created with no intermediate file:
$ kubectl -n tramontana create secret generic tramontana-secrets \
--from-literal=db_password="$(pass tramontana/db)" \
--dry-run=client -o yaml | kubectl apply -f -That --dry-run=client -o yaml | kubectl apply -f - is the idiomatic pattern for creating or updating a Secret without it ending up in the history or on disk.
# ~/tramontana-k8s/20-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: tramontana
namespace: tramontana
labels:
app: tramontana
spec:
replicas: 2
revisionHistoryLimit: 5 # how many old ReplicaSets to keep
selector:
matchLabels:
app: tramontana
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1 # one extra during the deployment
maxUnavailable: 0 # NEVER less capacity than declared
template:
metadata:
labels:
app: tramontana
version: "3.2.1"
spec:
# Spread the replicas across different nodes: if one node goes
# down, both of them do not go with it.
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app: tramontana
# --- Pod-level security (section 10) ---
securityContext:
runAsNonRoot: true
runAsUser: 997 # svc-tramontana, the usual uid
runAsGroup: 1002 # the tramontana group
fsGroup: 1002
seccompProfile:
type: RuntimeDefault
containers:
- name: tramontana
image: registry.tramontana.example/tramontana:3.2.1
# NEVER ':latest'. A moving tag makes two Pods of the same
# "version" run different code, and it makes a rollback roll
# back to nowhere.
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 8080
# --- Configuration from the ConfigMap ---
envFrom:
- configMapRef:
name: tramontana-config
env:
- name: TRAMONTANA_DB_PASSWORD
valueFrom:
secretKeyRef:
name: tramontana-secrets
key: db_password
# --- Resources: the relationship with the cgroups of 07-05 ---
# requests: what the SCHEDULER reserves in order to pick a node.
# limits: the real ceiling, imposed by cgroups v2.
# - Exceeding the MEMORY limit -> the kernel KILLS the
# container (OOMKilled): it is a hard limit.
# - Exceeding the CPU one -> it is THROTTLED, not killed:
# the process simply runs more slowly.
resources:
requests:
cpu: 100m # 0.1 of a core
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
# --- The three probes, each answering something different ---
# startupProbe: "has it finished starting?" While it fails,
# the other two do NOT run. It is what allows slow starts
# without relaxing liveness. 30 x 5 s = 150 s of headroom.
startupProbe:
httpGet: {path: /health, port: http}
periodSeconds: 5
failureThreshold: 30
# readinessProbe: "can it serve requests NOW?"
# If it fails, the Pod LEAVES the Service (it stops receiving
# traffic) but it is NOT restarted. It is the balancer's gate.
readinessProbe:
httpGet: {path: /health, port: http}
periodSeconds: 5
timeoutSeconds: 2
successThreshold: 1
failureThreshold: 3
# livenessProbe: "is it alive or hung?" If it fails, the
# kubelet KILLS AND RESTARTS the container. That is why it is
# the most dangerous: badly calibrated, it causes cascading
# restarts under load. LOOSER thresholds than readiness,
# always.
livenessProbe:
httpGet: {path: /health, port: http}
periodSeconds: 15
timeoutSeconds: 3
failureThreshold: 5
# --- Container-level security ---
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
# With readOnlyRootFilesystem you have to give explicit
# writable space for whatever genuinely needs it.
volumeMounts:
- name: tmp
mountPath: /tmp
- name: cache
mountPath: /var/cache/tramontana
# An orderly shutdown: it gives time to finish the requests in
# flight before the SIGTERM arrives.
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 5"]
terminationGracePeriodSeconds: 30
volumes:
- name: tmp
emptyDir: {sizeLimit: 64Mi}
- name: cache
emptyDir: {sizeLimit: 256Mi}# ~/tramontana-k8s/30-service.yaml
apiVersion: v1
kind: Service
metadata:
name: tramontana
namespace: tramontana
spec:
type: ClusterIP
selector:
app: tramontana # the Pod's labels, not the Deployment's
ports:
- name: http
port: 80 # the Service's port
targetPort: http # the container's named port
---
# The database lives OUTSIDE the cluster (srv-tramontana). This object
# gives it an internal name, so that the application always uses
# "postgresql" and not a hard-coded IP.
apiVersion: v1
kind: Service
metadata:
name: postgresql
namespace: tramontana
spec:
type: ExternalName
externalName: srv-tramontana.internal# ~/tramontana-k8s/40-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: tramontana
namespace: tramontana
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: websecure
traefik.ingress.kubernetes.io/router.tls: "true"
# The security headers from 08-01, as annotations
traefik.ingress.kubernetes.io/router.middlewares: tramontana-security@kubernetescrd
spec:
ingressClassName: traefik
tls:
- hosts: [bookings.tramontana.example]
secretName: tramontana-tls # a kubernetes.io/tls type Secret
rules:
- host: bookings.tramontana.example
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: tramontana
port: {name: http}
---
# A Traefik Middleware: the equivalent of the add_header lines of 08-01
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: security
namespace: tramontana
spec:
headers:
stsSeconds: 63072000
stsIncludeSubdomains: true
contentTypeNosniff: true
frameDeny: true
referrerPolicy: "strict-origin-when-cross-origin"# The certificate from 06-05, as a Secret
$ sudo kubectl -n tramontana create secret tls tramontana-tls \
--cert=/etc/letsencrypt/live/bookings.tramontana.example/fullchain.pem \
--key=/etc/letsencrypt/live/bookings.tramontana.example/privkey.pem
# Deploy everything, in order
$ kubectl apply -f ~/tramontana-k8s/
namespace/tramontana created
resourcequota/tramontana-quota created
configmap/tramontana-config created
deployment.apps/tramontana created
service/tramontana created
ingress.networking.k8s.io/tramontana created
$ kubectl -n tramontana get pods -o wide
NAME READY STATUS RESTARTS AGE NODE
tramontana-7d8f9c4b5-k2m4p 1/1 Running 0 48s srv-tramontana-test
tramontana-7d8f9c4b5-x9n7q 1/1 Running 0 48s k8s-node2
$ kubectl -n tramontana get endpoints tramontana
NAME ENDPOINTS AGE
tramontana 10.42.0.18:8080,10.42.1.22:8080 51sThe replicas have landed on different nodes thanks to topologySpreadConstraints, and the Service already has its two endpoints.
The three probes deserve a summary table, because confusing them is the most expensive mistake in this section:
| Probe | Question | If it fails | Risk if it is wrong |
|---|---|---|---|
startupProbe |
Has it finished starting? | Restarts once the headroom runs out | Without it, liveness kills slow starts |
readinessProbe |
Can it serve now? | Leaves the Service, no restart | Without it, traffic reaches a Pod that is not ready |
livenessProbe |
Is it hung? | Kills and restarts | Too aggressive: cascading restarts under load |
And the classic mistake, which turns up in real production systems with serious consequences: putting the same failureThreshold and periodSeconds on readiness and liveness. Under load the application answers a little more slowly, both fail, and the kubelet restarts Pods that were only busy — which reduces capacity, increases the load on the remaining ones and causes a cascading outage. Liveness always looser than readiness.
Rolling deployment and rollback
# 1. Deploy version 3.3.0. --record is deprecated; an annotation is
# used instead so that the reason is on the record.
$ kubectl -n tramontana set image deployment/tramontana \
tramontana=registry.tramontana.example/tramontana:3.3.0
$ kubectl -n tramontana annotate deployment/tramontana \
kubernetes.io/change-cause="Version 3.3.0: optimised reports"
# 2. Follow it live
$ kubectl -n tramontana rollout status deployment/tramontana
Waiting for deployment "tramontana" rollout to finish: 1 out of 2 new replicas have been updated...
Waiting for deployment "tramontana" rollout to finish: 1 old replicas are pending termination...
deployment "tramontana" successfully rolled out
# 3. During the process both ReplicaSets coexist
$ kubectl -n tramontana get rs
NAME DESIRED CURRENT READY AGE
tramontana-7d8f9c4b5 0 0 0 18m # 3.2.1
tramontana-9f2a1c8e7 2 2 2 42s # 3.3.0Zero requests lost, and the mechanism is the one you already know from 07-07: maxUnavailable: 0 guarantees that there is never less capacity than declared, and the readinessProbe guarantees that a Pod receives no traffic until it answers. The draining you did in 07-07 with socat against HAProxy's socket is, here, a property of the system.
# 4. The new version fails in production: roll back
$ kubectl -n tramontana rollout history deployment/tramontana
REVISION CHANGE-CAUSE
1 Initial version 3.2.1
2 Version 3.3.0: optimised reports
$ kubectl -n tramontana rollout undo deployment/tramontana
deployment.apps/tramontana rolled back
$ kubectl -n tramontana rollout status deployment/tramontana
deployment "tramontana" successfully rolled outThe rollback took eleven seconds, against the minutes of the deploy.sh from 04-07 with its ln -sfn. It is the strongest argument in favour of Kubernetes: reversibility is a property of the system, not a script you have to remember to write correctly.
With the caveat that always has to be stated: rollout undo reverts the code, not the data. If version 3.3.0 ran a schema migration, going back to 3.2.1 leaves the old application against a new database. It is the same problem as in 04-07, and the solution is the same: backwards-compatible migrations. Kubernetes does not solve it.
# 5. Scaling, which is trivial and that is why it catches the eye
$ kubectl -n tramontana scale deployment/tramontana --replicas=4
$ kubectl -n tramontana get pods --no-headers | wc -l
4Security: securityContext, NetworkPolicy, RBAC and PSS
securityContext
Kubernetes' defaults are permissive: without a securityContext, a container runs as root with a broad set of capabilities. Everything from 07-05 applies here declaratively:
| Directive | What it prevents | Equivalent in 07-05 |
|---|---|---|
runAsNonRoot: true |
The container running as root | USER in the Dockerfile |
runAsUser: 997 |
— | --user |
allowPrivilegeEscalation: false |
Gaining privileges via SUID | --security-opt no-new-privileges |
readOnlyRootFilesystem: true |
Writing to the file system | --read-only |
capabilities.drop: ["ALL"] |
Every kernel capability | --cap-drop=ALL |
seccompProfile: RuntimeDefault |
Dangerous system calls | --security-opt seccomp |
readOnlyRootFilesystem: true is the one that frustrates the most attacks in practice: most exploits need to write something to disk in order to persist or to download the next stage.
# Check that it really is applied
$ kubectl -n tramontana exec deploy/tramontana -- id
uid=997 gid=1002 groups=1002
$ kubectl -n tramontana exec deploy/tramontana -- touch /test
touch: /test: Read-only file system
command terminated with exit code 1NetworkPolicy
By default, every Pod in the cluster can talk to every other one. It is a flat network, and in a cluster with several applications that means a compromised Pod reaches any other.
# ~/tramontana-k8s/50-networkpolicy.yaml
# 1. Deny EVERYTHING by default in the namespace (allowlist, 06-03)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-all
namespace: tramontana
spec:
podSelector: {} # every Pod in the namespace
policyTypes: [Ingress, Egress]
---
# 2. Allow only what is needed
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-tramontana
namespace: tramontana
spec:
podSelector:
matchLabels: {app: tramontana}
policyTypes: [Ingress, Egress]
ingress:
# Only from the Ingress (Traefik lives in kube-system)
- from:
- namespaceSelector:
matchLabels: {kubernetes.io/metadata.name: kube-system}
ports: [{protocol: TCP, port: 8080}]
egress:
# DNS: without this, NOTHING works and the failure is baffling
- to:
- namespaceSelector:
matchLabels: {kubernetes.io/metadata.name: kube-system}
ports:
- {protocol: UDP, port: 53}
- {protocol: TCP, port: 53}
# PostgreSQL, outside the cluster
- to:
- ipBlock: {cidr: 10.0.2.15/32}
ports: [{protocol: TCP, port: 6432}]Forgetting the DNS rule is the number one mistake with NetworkPolicy. The policy is applied, everything stops working, and the symptom — "the name cannot be resolved" — does not point at the firewall. With policyTypes: [Egress], traffic to CoreDNS is blocked too.
An important warning for k3s: the default CNI (Flannel) does not implement NetworkPolicy, so these objects are accepted and do nothing. You have to install k3s with --flannel-backend=none and deploy Calico, or use --disable-network-policy=false depending on the version. Checking it is essential: believing you have segmentation when you do not is worse than not having it.
RBAC, briefly
Role-based access control uses four objects:
| Object | Scope | What it defines |
|---|---|---|
Role |
One namespace | Which verbs on which resources |
ClusterRole |
The whole cluster | The same, globally |
RoleBinding |
One namespace | Who has that Role |
ClusterRoleBinding |
The whole cluster | Who has that ClusterRole |
# Luis: read-only and logs in his namespace. He cannot see Secrets.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: tramontana
name: reader
rules:
- apiGroups: ["", "apps"]
resources: ["pods", "pods/log", "deployments", "services", "configmaps"]
verbs: ["get", "list", "watch"]
# 'secrets' does NOT appear: reading them is reading the passwords (base64)# Check permissions, your own or somebody else's
$ kubectl auth can-i get secrets -n tramontana --as=luis
no
$ kubectl auth can-i get pods -n tramontana --as=luis
yeskubectl auth can-i is the RBAC verification tool, and it is the way to check that what is forbidden is forbidden, just as you did with the PostgreSQL roles in 08-02.
Pod Security Standards
They replace PodSecurityPolicy, withdrawn in 1.25. They are three profiles applied per namespace with labels:
| Profile | What it allows |
|---|---|
privileged |
Everything. No restrictions |
baseline |
Blocks the most dangerous things: privileged, hostNetwork, hostPath |
restricted |
Requires runAsNonRoot, drop ALL, seccomp, no escalation |
The namespace's enforce: restricted makes the apiserver reject any Pod that does not comply. It is a safety net that does not depend on remembering to add the securityContext:
$ kubectl -n tramontana run bad --image=nginx --privileged
Error from server (Forbidden): pods "bad" is forbidden: violates PodSecurity
"restricted:latest": privileged (container "bad" must not set
securityContext.privileged=true), allowPrivilegeEscalation != false,
unrestricted capabilities, runAsNonRoot != true, seccompProfileDiagnosis: reading events and the four common failures
The first command for any problem is kubectl describe, and what matters is at the bottom, in Events.
$ kubectl -n tramontana describe pod tramontana-7d8f9c4b5-k2m4p | tail -12
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m default-scheduler Successfully assigned...
Normal Pulling 2m kubelet Pulling image "...:3.2.1"
Normal Pulled 1m kubelet Successfully pulled image
Normal Created 1m kubelet Created container tramontana
Normal Started 1m kubelet Started container tramontana
Warning Unhealthy 30s (x3 over 40s) kubelet Readiness probe failed:
Get "http://10.42.0.18:8080/health": dial tcp: connect: connection refusedThat last line explains the whole problem: the application is not listening on 8080 yet.
The four failures you will genuinely see
1. CrashLoopBackOff — the container starts, dies, and Kubernetes waits longer and longer before retrying (10 s, 20 s, 40 s... up to 5 min).
$ kubectl -n tramontana get pods
NAME READY STATUS RESTARTS AGE
tramontana-9f2a1c8e7-p4k9m 0/1 CrashLoopBackOff 5 (48s ago) 4m
# The key: --previous. Without it you see the logs of the CURRENT
# attempt, which has just started and is empty.
$ kubectl -n tramontana logs tramontana-9f2a1c8e7-p4k9m --previous
FATAL: could not connect to the database: password authentication
failed for user "svc_tramontana"
# Confirm the cause
$ kubectl -n tramontana get secret tramontana-secrets -o jsonpath='{.data.db_password}' \
| base64 -d | head -c8; echo '...'
PLACEHO...There it is: the Secret has the manifest's placeholder, not the real password.
Cause of CrashLoopBackOff |
How it is confirmed |
|---|---|
| A configuration or credentials error | logs --previous |
| A missing dependency (an unreachable DB) | logs --previous + exec into another Pod |
| The container's command finishes | describe: Exit Code: 0 |
A livenessProbe that is too aggressive |
describe: Unhealthy events before each restart |
| OOMKilled | describe: Reason: OOMKilled |
2. ImagePullBackOff / ErrImagePull
$ kubectl -n tramontana describe pod tramontana-x | grep -A3 Events
Warning Failed 30s kubelet Failed to pull image
"registry.tramontana.example/tramontana:3.3.1": failed to resolve reference:
unexpected status: 401 Unauthorized| Message | Cause | Solution |
|---|---|---|
401 Unauthorized |
A missing registry credential | imagePullSecrets |
not found / manifest unknown |
The tag does not exist (a typo) | Check with crane ls or skopeo |
no such host |
The node's DNS does not resolve the registry | Check on the node, not in the Pod |
connection refused |
The registry is down, or a firewall | Check from the node |
3. Pending — the Pod exists and no node accepts it.
$ kubectl -n tramontana describe pod tramontana-y | tail -4
Warning FailedScheduling 20s default-scheduler
0/2 nodes are available: 1 Insufficient cpu, 1 Insufficient memory.
preemption: 0/2 nodes are available: 2 No preemption victims found.
# How much is actually committed on each node
$ kubectl describe node k8s-node2 | grep -A6 'Allocated resources'
Allocated resources:
Resource Requests Limits
cpu 1750m (87%) 3200m (160%)
memory 1894Mi (94%) 2560Mi (128%)The scheduler reserves according to requests, not to actual usage. A node with 20 % of its CPU in use can reject Pods if its requests add up to 95 %. It is the most common confusion with Pending, and it is why setting generous requests "just in case" wastes the whole cluster.
Other causes of Pending: a PVC that cannot be provisioned, a nodeSelector that matches no node, or taints without the corresponding toleration.
4. OOMKilled — the kernel killed the container for exceeding the memory limit.
$ kubectl -n tramontana describe pod tramontana-z | grep -A4 'Last State'
Last State: Terminated
Reason: OOMKilled
Exit Code: 137 # 128 + 9 (SIGKILL)It is the same cgroups mechanism as 07-05. Exit Code: 137 is the unmistakable signature. And the decision is not automatic: you have to work out whether the limit is too low or whether there is a memory leak in the application — raising the limit without looking only delays the problem.
The diagnostic toolbox
# The namespace's events, in chronological order
$ kubectl -n tramontana get events --sort-by=.lastTimestamp | tail -10
# Real consumption (it needs metrics-server, which k3s ships)
$ kubectl -n tramontana top pods
NAME CPU(cores) MEMORY(bytes)
tramontana-7d8f9c4b5-k2m4p 12m 184Mi
# Debug the network from inside the cluster, without touching the real Pods
$ kubectl -n tramontana run debug --rm -it --image=nicolaka/netshoot \
--restart=Never -- bash
debug:~# nslookup tramontana
debug:~# curl -sv http://tramontana/health
debug:~# nc -zv 10.0.2.15 6432
# Forward a port to your laptop, without exposing anything
$ kubectl -n tramontana port-forward svc/tramontana 8080:80
$ curl -s localhost:8080/healthport-forward is the most used day-to-day tool: it lets you reach an internal service from your laptop without creating an Ingress or a NodePort.
Automation with Ansible
There is an obvious temptation here and it is worth avoiding: Ansible must not replace kubectl apply. The manifests are already declarative and idempotent; wrapping them in Ansible adds a layer without gaining anything. The correct division:
| Layer | Tool | Why |
|---|---|---|
| Preparing the node's operating system | Ansible | It is machine configuration |
| Installing and configuring k3s | Ansible | It is a systemd service |
| Joining nodes to the cluster | Ansible | It needs the token, which is a secret |
| Deploying applications | kubectl apply or Helm |
It is already declarative |
| Keeping the cluster in sync with Git | Argo CD or Flux (GitOps) | It is the industry pattern |
# ~/tramontana-infra/roles/k3s/defaults/main.yml
---
k3s_version: "v1.30.4+k3s1"
k3s_url: "https://192.168.122.104:6443"
k3s_datastore: sqlite
k3s_disable: []
k3s_extra_args_server: "--write-kubeconfig-mode 0644 --tls-san {{ ansible_default_ipv4.address }}"# ~/tramontana-infra/roles/k3s/tasks/main.yml
---
- name: Safety guard — NEVER in production
ansible.builtin.assert:
that: inventory_hostname != 'srv-tramontana'
fail_msg: >-
This role must NOT be run on srv-tramontana. k3s rewrites the
nftables rules and starts its own container runtime; on the
production server it would interfere with ufw, AppArmor and the
tramontana service.
- name: System requirements
ansible.builtin.apt:
name: [curl, iptables, apparmor-utils]
state: present
- name: Download the k3s installer
ansible.builtin.get_url:
url: https://get.k3s.io
dest: /usr/local/src/k3s-install.sh
mode: '0755'
- name: Install k3s on the SERVER node
ansible.builtin.command:
cmd: /usr/local/src/k3s-install.sh
creates: /usr/local/bin/k3s # idempotence (07-06)
environment:
INSTALL_K3S_VERSION: "{{ k3s_version }}"
INSTALL_K3S_EXEC: "server {{ k3s_extra_args_server }}"
when: "'k3s_server' in group_names"
- name: Read the token from the server node
ansible.builtin.slurp:
src: /var/lib/rancher/k3s/server/node-token
register: k3s_token_raw
when: "'k3s_server' in group_names"
- name: Share the token with the agents
ansible.builtin.set_fact:
k3s_token: "{{ hostvars[groups['k3s_server'][0]]['k3s_token_raw']['content'] | b64decode | trim }}"
no_log: true
- name: Install k3s on the AGENT nodes
ansible.builtin.command:
cmd: /usr/local/src/k3s-install.sh
creates: /usr/local/bin/k3s
environment:
INSTALL_K3S_VERSION: "{{ k3s_version }}"
K3S_URL: "{{ k3s_url }}"
K3S_TOKEN: "{{ k3s_token }}"
INSTALL_K3S_EXEC: "agent"
no_log: true
when: "'k3s_agent' in group_names"
- name: Wait for the node to be Ready
ansible.builtin.command: "k3s kubectl get node {{ ansible_hostname }} -o json"
register: node
until: >-
(node.stdout | from_json).status.conditions
| selectattr('type','eq','Ready') | map(attribute='status') | first == 'True'
retries: 30
delay: 10
changed_when: false
delegate_to: "{{ groups['k3s_server'][0] }}"
# --- A backup of the cluster, which is what really matters ---
- name: Install the etcd/SQLite backup script
ansible.builtin.copy:
src: k3s_backup.sh
dest: /usr/local/bin/k3s_backup.sh
mode: '0750'
when: "'k3s_server' in group_names"
- name: Daily timer for backing up the cluster state
ansible.builtin.copy:
src: "{{ item }}"
dest: "/etc/systemd/system/{{ item }}"
mode: '0644'
loop: [k3s-backup.service, k3s-backup.timer]
notify: Reload systemd
when: "'k3s_server' in group_names"That initial assert is the literal application of the warning in section 1, turned into code that cannot be ignored by carelessness. And backing up the cluster state is not optional: without etcd or its SQLite, the cluster does not exist.
# roles/k3s/files/k3s_backup.sh (the core)
$ k3s etcd-snapshot save --name daily # with etcd
# or, with SQLite:
$ sqlite3 /var/lib/rancher/k3s/server/db/state.db ".backup '/srv/backups/k3s.db'"
$ restic backup /srv/backups/k3s.db --tag k3sThe real operational complexity
Here is the part Kubernetes presentations leave out, and the one that really answers the question in section 2.
1. The upgrades are constant. Kubernetes releases three minor versions a year and each one gets around fourteen months of support. That forces you to upgrade the cluster at least twice a year, indefinitely. Each upgrade means reading the release notes looking for breaking changes, upgrading the control plane and then the nodes, and verifying that the workloads still work.
2. API versions get withdrawn, and they break manifests that used to work.
| Resource | Old API | Current API | Withdrawn in |
|---|---|---|---|
| Ingress | extensions/v1beta1 |
networking.k8s.io/v1 |
1.22 |
| CronJob | batch/v1beta1 |
batch/v1 |
1.25 |
| PodSecurityPolicy | policy/v1beta1 |
Removed: use PSS | 1.25 |
| HorizontalPodAutoscaler | autoscaling/v2beta2 |
autoscaling/v2 |
1.26 |
A manifest written two years ago can fail after an upgrade. There are tools (pluto, kubent) to detect it beforehand, and using them is mandatory.
3. etcd is delicate. It is a distributed database with quorum: it needs 3 or 5 nodes (an odd number), it is sensitive to disk latency, and it requires periodic compaction and defragmentation. A corrupted etcd with no backup is a lost cluster. And everything about quorum from 07-07 applies.
4. The internal certificates expire. Kubernetes uses mutual TLS between all of its components, with a PKI of its own and certificates that expire after a year. kubeadm renews them on upgrade, but a cluster that has not been touched for fourteen months stops working all at once, with baffling authentication errors.
$ sudo kubeadm certs check-expiration | head -5
CERTIFICATE EXPIRES RESIDUAL TIME
apiserver Aug 18, 2027 13:02 UTC 364d
etcd-server Aug 18, 2027 13:02 UTC 364d5. The ecosystem moves very fast. Docker Shim withdrawn in 1.24, PSP replaced by PSS, Ingress evolving towards the Gateway API. What you learn goes out of date faster than in any other area of the system.
6. Diagnosing demands understanding more layers. A failure can be in the application, the container, the Pod, the Service, the CNI, the Ingress, the DNS, the scheduler or the kubelet. Everything from Module 7 is still necessary; Kubernetes adds layers, it does not replace them.
The estimated annual cost, in hours, which is the figure you have to take into a decision:
| Activity | Hours/year |
|---|---|
| Two minor version upgrades | 16-32 |
| Maintaining etcd and its backups | 8-12 |
| Updating manifests for withdrawn APIs | 4-8 |
| Diagnosing cluster-specific incidents | 20-40 |
| Continuous training | 20-40 |
| Total | 70-130 h/year |
Between two and four weeks of work a year, just to keep the cluster alive, without counting deploying applications. With a managed service, half that. With systemd and Ansible on three machines, practically zero.
That number is the final answer to the question in section 2, and it also explains why managed Kubernetes is so successful: for most companies, 100 € a month is cheaper than 100 hours a year.
Common Mistakes and Tips
- Using
:latestfor the image. Two Pods of the same "version" can run different code, androllout undorolls back to nowhere. Immutable tags, always. - Creating loose Pods instead of Deployments. Nobody watches them: if they die, they do not come back.
- A
livenessProbeequal to or stricter than thereadinessProbe. Under load, healthy Pods get restarted and the service falls over in a cascade. Liveness always looser. - Not putting a
startupProbeon slow-starting applications. Liveness kills the container before it finishes starting, in a loop. - Believing a Secret is encrypted. It is base64.
kubectl get secret -o jsonpath | base64 -dand there it is. - Pushing Secrets to Git. The YAML with
data:is the file with the password. Sealed Secrets or an external store. - Forgetting the DNS rule in an Egress NetworkPolicy. Everything stops working and the symptom does not point at the firewall.
- Applying NetworkPolicy with a CNI that does not implement it. Flannel in k3s accepts them and ignores them: you think you have segmentation and you do not.
requestsequal tolimitsand very generous. The scheduler reserves byrequests: you waste the cluster and causePendingwith empty nodes.- Raising the memory limit on seeing an
OOMKilledwithout investigating. It could be a leak, and you are only delaying the problem. kubectl logswithout--previouson aCrashLoopBackOff. You see the logs of the current attempt, which is empty.- Not backing up etcd or the SQLite. Without it, the cluster cannot be rebuilt.
- Ignoring the expiry of the internal certificates. A cluster untouched for fourteen months stops working all at once.
- Installing Kubernetes on an existing production server. It rewrites network rules and starts its own runtime. A dedicated machine.
- Setting up Kubernetes for three services. It is the architectural mistake of this entire lesson: 70-130 h/year to solve problems you do not have.
- A tip on method. Faced with any failure:
kubectl describeand read theEventsfrom the bottom up. It solves most problems before you touch anything.
Exercises
Exercise 1
After deploying version 3.3.0, the Pods go into CrashLoopBackOff and the service goes down. Diagnose the problem methodically, restore the service and explain what would have prevented the incident.
Exercise 2
Compare, for Tramontana's case, the Kubernetes architecture from this lesson with option B recommended in 07-07, using objective criteria, and issue a recommendation.
Exercise 3
Write a CronJob manifest that runs the database backup inside the cluster, with the same guarantees as tramontana-backup.timer from 05-05, and explain what is gained and what is lost compared with the systemd timer.
Solutions
Solution 1
Step 1: the overall picture, before touching anything.
$ kubectl -n tramontana get pods
NAME READY STATUS RESTARTS AGE
tramontana-7d8f9c4b5-k2m4p 1/1 Running 0 3h # 3.2.1
tramontana-9f2a1c8e7-p4k9m 0/1 CrashLoopBackOff 4 (52s ago) 3m # 3.3.0First important and reassuring fact: thanks to maxUnavailable: 0, the version 3.2.1 Pod is still running and serving traffic. The deployment is blocked, not down. That takes the pressure off and allows you to diagnose without rushing.
$ kubectl -n tramontana rollout status deployment/tramontana --timeout=10s
error: timed out waiting for the condition
$ kubectl -n tramontana get endpoints tramontana
NAME ENDPOINTS AGE
tramontana 10.42.0.18:8080 3hA single endpoint: the new Pod has never reached Ready, so the Service has never sent it traffic. The readinessProbe has done exactly its job.
Step 2: the logs of the previous attempt.
$ kubectl -n tramontana logs tramontana-9f2a1c8e7-p4k9m --previous
[2026-08-18T16:02:11Z] INFO Tramontana Bookings 3.3.0 starting
[2026-08-18T16:02:11Z] INFO reading configuration from the environment
[2026-08-18T16:02:11Z] ERROR variable TRAMONTANA_CACHE_URL not defined
[2026-08-18T16:02:11Z] FATAL incomplete configuration; abortingThe root cause in four lines. Version 3.3.0 introduces a new configuration variable — TRAMONTANA_CACHE_URL, for the report cache — that is not in the ConfigMap. The application fails fast and clearly, which is the correct behaviour.
Step 3: confirm and rule out other causes.
$ kubectl -n tramontana describe pod tramontana-9f2a1c8e7-p4k9m | \
grep -A6 'Last State'
Last State: Terminated
Reason: Error
Exit Code: 1 # NOT 137: it is not OOMKilled
Started: Tue, 18 Aug 2026 16:02:11 +0200
Finished: Tue, 18 Aug 2026 16:02:11 +0200
$ kubectl -n tramontana get configmap tramontana-config -o jsonpath='{.data}' | \
jq 'keys'
["db_host","db_name","db_port","listen","log_level","max_connections",
"port","query_timeout"]Exit Code: 1 with the start and the stop in the same second rules out OOMKilled (137), badly calibrated probes (it would have reached Running for a while) and network problems (it never got as far as connecting to anything). And the ConfigMap confirms the missing key.
Step 4: decide. And the correct decision is to roll back first.
$ kubectl -n tramontana rollout undo deployment/tramontana
deployment.apps/tramontana rolled back
$ kubectl -n tramontana rollout status deployment/tramontana
deployment "tramontana" successfully rolled outWhy roll back before fixing, even though the fix looks trivial: the Deployment is in an inconsistent state, with a new ReplicaSet trying to start in a loop every few seconds. Each retry generates events, consumes resources and pollutes the diagnosis. Leaving the system in a known, stable state before correcting it is the discipline that stops you making things worse — and it is the same logic as the decision tree in the recovery runbook from 07-06.
Step 5: the fix, tested before applying it.
$ kubectl -n tramontana patch configmap tramontana-config \
--type merge -p '{"data":{"cache_url":"redis://cache.tramontana.svc:6379"}}'# 20-deployment.yaml — the explicit mapping of the variable
env:
- name: TRAMONTANA_CACHE_URL
valueFrom:
configMapKeyRef:
name: tramontana-config
key: cache_url# Verify in a throwaway Pod BEFORE touching the Deployment
$ kubectl -n tramontana run verify --rm -it --restart=Never \
--image=registry.tramontana.example/tramontana:3.3.0 \
--overrides='{"spec":{"containers":[{"name":"verify",
"image":"registry.tramontana.example/tramontana:3.3.0",
"envFrom":[{"configMapRef":{"name":"tramontana-config"}}],
"env":[{"name":"TRAMONTANA_CACHE_URL","valueFrom":
{"configMapKeyRef":{"name":"tramontana-config","key":"cache_url"}}}]}]}}' \
-- /usr/bin/tramontana --check-config
configuration valid# Now yes
$ kubectl apply -f ~/tramontana-k8s/20-deployment.yaml
$ kubectl -n tramontana rollout status deployment/tramontana
deployment "tramontana" successfully rolled out
$ kubectl -n tramontana get pods
NAME READY STATUS RESTARTS AGE
tramontana-3c7e9a2b1-h4j8k 1/1 Running 0 42s
tramontana-3c7e9a2b1-w2n5r 1/1 Running 0 28sOne detail worth knowing: changing a ConfigMap does not restart the Pods that use it as environment variables. If that had been needed:
What would have prevented the incident, which is the question that really matters:
| Measure | How it would have prevented it |
|---|---|
| Testing the manifest in a test namespace | The failure would have happened where it does not matter |
| Validating the configuration when the image starts | A --check-config in the build fails before deploying |
| Release notes with the configuration changes | Luis must document every new variable |
| A default value in the application | A new variable with a reasonable value breaks nothing |
| CI that applies the manifests to an ephemeral cluster | The failure is caught in the code review |
An alert on the first CrashLoopBackOff |
It does not prevent it, but it detects it in a minute |
And the three lessons of method:
maxUnavailable: 0turned an outage into a blocked deployment. It is the line of configuration with the best benefit/cost ratio in the whole manifest, and it deserves to be in any production Deployment.- The
readinessProbestopped the Service sending traffic to a broken Pod. Without it, half the requests would have failed for three minutes. --previouswas what gave the answer. Without that option,kubectl logsreturned a freshly started, empty container, and the diagnosis would have taken far longer.
Solution 2
A comparison with objective criteria, on Tramontana's real case: a monolithic application, one database, ~500 bookings a month, two technical people.
| Criterion | Kubernetes (k3s) | Option B from 07-07 (2 nodes + HAProxy) |
|---|---|---|
| Machines needed | 3 (control plane with quorum) or 2 with a SPOF | 3 (2 app + 1 load balancer) |
| Base RAM consumed | 812 MiB for the cluster alone | ~120 MiB (HAProxy + keepalived) |
| Deployment without interruption | Native, RollingUpdate |
Manual draining with socat, a script |
| Rollback | rollout undo, 11 s |
ln -sfn + restart, ~2 min |
| Self-healing of a Pod | Automatic | systemd's Restart=on-failure |
| Self-healing when a node goes down | Automatic, it relocates | HAProxy takes it out; it does not relocate |
| Scaling to 4 replicas | One command, seconds | Provision a machine + Ansible, ~1 h |
| Learning curve | 3-6 months | You already have it |
| Annual maintenance | 70-130 h | ~15 h |
| New pieces to master | etcd, CNI, Ingress, RBAC, PSS, CRDs | None |
| Diagnosing a failure | More layers to rule out | The usual ones |
| Measured full rebuild | Unmeasured; the cluster adds steps | 50 min, measured |
| Achievable availability | ~99.8-99.9 % | ~99.8 % |
| Estimated annual cost | 3 VMs + 100 h of work | 3 VMs + 15 h |
| Transferable to another job | A great deal | Moderate |
The four points where Kubernetes genuinely wins, without exaggerating:
- Deployment and rollback. Eleven seconds against two minutes, and with no script of your own to maintain. It is a real difference.
- Relocation when a node goes down. HAProxy takes the failed node out of the rotation but does not recover the capacity; Kubernetes starts the missing replicas on the surviving node.
- Scaling. One command against an hour of work. It only matters if scaling is frequent, and at Tramontana it is not.
- Employability. It is a legitimate personal argument but it is not an architectural argument for the company, and they have to be separated honestly.
The five where it loses:
- 812 MiB of RAM before deploying anything, on 3.8 GB machines. That is more than 20 % of the server devoted to orchestration.
- 70-130 hours a year of indefinite maintenance, against about 15.
- Three to six months before you can resolve incidents fluently. During that period, reliability goes down.
- More failure modes. It is the same argument as 07-07: a system with more pieces has more ways of breaking, and Kubernetes adds a lot of them.
- The database is still outside. The hard problem from 07-07 — state — is not solved by Kubernetes. PostgreSQL would still be on
srv-tramontana, with its manual promotion and its RPO.
The analysis that settles the question, and it is the same reasoning as in 07-07: both options reach roughly the same availability, around 99.8 %, because most of Tramontana's interruptions do not come from a machine failing but from deployments and maintenance, and both of them solve those. Kubernetes does it more elegantly and automatically; option B does it with tools the team already masters.
When two options give the same result, the simpler one wins. And the difference in maintenance — between 55 and 115 hours a year — is equivalent to two or three weeks of work that would not be spent improving the product.
Recommendation: keep option B from 07-07 for production.
And keep the k3s cluster on
srv-tramontana-testas a learning lab, at zero marginal cost, because the machine already exists. It is where to practise without risk and where to be ready for the scenario that would change the recommendation.
The three circumstances that would change it, written down in advance so that they can be recognised:
| Change | Why it would change the decision |
|---|---|
| Tramontana splits into 5+ services | Orchestration starts to pay for itself |
| Managed Kubernetes is contracted in the cloud | 60 % of the maintenance disappears |
| Somebody with real experience joins | The learning curve stops being a cost |
And a final warning worth putting in writing: today's correct decision may not be correct in two years, and that does not mean it is badly taken today. Setting an annual review is more professional than choosing the architecture you might need one day.
Solution 3
# ~/tramontana-k8s/60-cronjob-backup.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
name: database-backup
namespace: tramontana
spec:
# 02:30, the same as tramontana-backup.timer (05-05).
# CAREFUL: the schedule is interpreted in the CLUSTER'S TIME ZONE,
# which is UTC by default. timeZone (stable since 1.27) solves it;
# without it, the backup would run at 04:30 in summer.
schedule: "30 2 * * *"
timeZone: "Europe/Madrid"
# The equivalent of the flock from 04-07: if yesterday's backup is
# still running, another one is NOT launched on top of it.
concurrencyPolicy: Forbid
# If the cluster was down at 02:30, does it run when it comes back?
# 3600 = only if less than 1 h has passed. It is the equivalent of
# the timer's Persistent=true, but BOUNDED: without this field, a
# badly delayed Job would be launched at an inconvenient moment.
startingDeadlineSeconds: 3600
successfulJobsHistoryLimit: 7 # keep 7 successful runs
failedJobsHistoryLimit: 14 # and 14 failed ones: those are the ones you look at
jobTemplate:
spec:
# Retries on failure, with exponential backoff
backoffLimit: 2
# An absolute ceiling: a hung backup does not block the next one
activeDeadlineSeconds: 3600
# Automatic cleanup of the Job 3 days later
ttlSecondsAfterFinished: 259200
template:
spec:
restartPolicy: OnFailure
securityContext:
runAsNonRoot: true
runAsUser: 997
runAsGroup: 1002
fsGroup: 1002
seccompProfile: {type: RuntimeDefault}
containers:
- name: backup
image: registry.tramontana.example/backup:1.4.0
command: ["/usr/local/bin/backup_tramontana.sh"]
env:
- name: PGHOST
valueFrom:
configMapKeyRef: {name: tramontana-config, key: db_host}
- name: PGDATABASE
valueFrom:
configMapKeyRef: {name: tramontana-config, key: db_name}
- name: PGPASSWORD
valueFrom:
secretKeyRef: {name: tramontana-secrets, key: db_password}
- name: RESTIC_REPOSITORY
valueFrom:
secretKeyRef: {name: restic-secrets, key: repository}
- name: RESTIC_PASSWORD
valueFrom:
secretKeyRef: {name: restic-secrets, key: password}
resources:
requests: {cpu: 100m, memory: 256Mi}
# The backup compresses: it needs more headroom than the app
limits: {cpu: "1", memory: 1Gi}
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: {drop: ["ALL"]}
volumeMounts:
- {name: work, mountPath: /tmp}
- {name: restic-cache, mountPath: /var/cache/restic}
volumes:
- name: work
emptyDir: {sizeLimit: 4Gi}
# The restic cache speeds up successive backups a great
# deal: persisting it between runs is a real optimisation.
- name: restic-cache
persistentVolumeClaim: {claimName: restic-cache}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: restic-cache
namespace: tramontana
spec:
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 2Gi}}# Test it WITHOUT waiting until 02:30 (the equivalent of systemctl start)
$ kubectl -n tramontana create job --from=cronjob/database-backup manual-test
$ kubectl -n tramontana logs job/manual-test -f
[2026-08-18 17:12:03] starting the PostgreSQL backup
[2026-08-18 17:12:41] base backup verified: 1.7 GiB
[2026-08-18 17:14:12] restic: snapshot a3f19c8d saved
[2026-08-18 17:14:20] backup completed
$ kubectl -n tramontana get cronjob
NAME SCHEDULE TIMEZONE SUSPEND ACTIVE LAST SCHEDULE
database-backup 30 2 * * * Europe/Madrid False 0 8hWhat is gained and what is lost, which is the central part of the exercise:
| Kubernetes CronJob | tramontana-backup.timer (systemd) |
|
|---|---|---|
| Runs if a node goes down | Yes, on another node | No: it dies with the machine |
| Time zone | timeZone (1.27+); UTC by default |
The system's zone, naturally |
| Mutual exclusion | concurrencyPolicy: Forbid |
flock (04-07) |
| Delayed execution | startingDeadlineSeconds |
Persistent=true |
| Run history | Job objects you can query | journalctl -u |
| Retries | backoffLimit, native |
Restart= + RestartSec |
| Resource isolation | requests/limits |
CPUQuota, MemoryMax |
| Secrets | Secret in base64 | systemd-creds, encrypted (06-05) |
| Failure notification | Requires external monitoring | OnFailure=, native |
| Access to the host's disk | Complicated, and rightly so | Direct |
| Debugging | kubectl logs job/... |
journalctl, or run the script |
| Pieces needed | Cluster + registry + image | A script and a .timer file |
What is genuinely gained is two things, and only two:
- Resistance to a node failure. It is the strong argument:
tramontana-backup.timerlives onsrv-tramontanaand if that machine is switched off at 02:30, there is no backup that night and nobody finds out untilcheck_backup.shdetects it at 08:00. The CronJob runs on any available node. - Real isolation and limits, just as easily. Although
systemdalso gives them withMemoryMax, here they come as standard.
What is lost, and it carries weight:
- The secrets get worse.
systemd-credsfrom 06-05 stores them encrypted with the TPM or a host key; a Kubernetes Secret is base64 in etcd. It is an objective step backwards, unless encryption at rest or Vault is added. - Failure notification stops being native.
OnFailure=in systemd sends the alert directly; with a CronJob you have to monitorkube_job_status_failedfrom Prometheus (08-06). One more piece. - Reaching the host's disk gets complicated, and rightly so: mounting
/srv/tramontana/backupsin a Pod requires ahostPath, which therestrictedPod Security Standards forbid. The flow would have to be rewritten so that the backup goes straight toresticwithout passing through the local disk, which is cleaner but is work. - An image has to be built and published with the script,
pg_dump,resticand their dependencies — and kept up to date. The timer uses what is already installed.
A recommendation consistent with solution 2:
Keep
tramontana-backup.timerin systemd. The only real benefit — resistance to a node failure — can be achieved without Kubernetes: by running the timer on the second application node too with a shared lock, or by launching it from a machine other than the one being backed up, which is also better design because the backup does not depend on the backed-up server being alive.And there is one additional argument: the backup must carry on working when the cluster fails. A backup mechanism that depends on the infrastructure it backs up is exactly the kind of coupling that has to be avoided. It is the same principle by which the runbook from 05-08 lives outside the server and the AIDE database outside the machine.
Conclusion
You have built a two-node Kubernetes cluster, you have deployed Tramontana Bookings on top of it with complete manifests, you have carried out a rolling deployment without losing a single request and a rollback in eleven seconds. And you know what is underneath: the apiserver as the only way in, etcd holding all the state, the scheduler choosing a node according to requests, the controllers tirelessly reconciling the real with the declared, and the kubelet running on each node whatever it is given. You know that if the control plane goes down, the workload carries on working — and why.
You understand the objects in their dependency order and why each one exists. That nobody watches a loose Pod. That the Deployment manages ReplicaSets and that rollout undo lives there. That a Service gives a stable name and IP to a moving target, and that the internal DNS makes it transparent. That an Ingress is the Nginx from 08-01 written as a declarative object — in fact, the most widely used controller is Nginx underneath. And you know that a Secret is base64, not encryption, which links directly to pass and systemd-creds from 06-05 and explains why Sealed Secrets and Vault exist.
You have mastered the three probes and the difference that separates them, which is what causes the most incidents in real production systems: readiness takes you out of the rotation, liveness kills and restarts, startup gives the start-up headroom — and liveness always looser than readiness, or cascading restarts under load are only a matter of time. You know that requests is what the scheduler reserves and limits what the cgroup imposes, with the key asymmetry: exceeding the memory kills the container, exceeding the CPU only throttles it. And you know how to read a CrashLoopBackOff with --previous, a Pending by looking at the node's committed requests, and an Exit Code: 137 as the OOM killer's signature.
But the most valuable thing in this lesson is the conclusion it started with: for Tramontana, Kubernetes is disproportionate, and now you can defend that with numbers — 812 MiB of RAM before deploying anything, 70 to 130 hours of maintenance a year, three to six months of learning curve — instead of with an intuition. That the professional answer to "should we use Kubernetes?" can be "no, and here is why" is exactly the same criterion with which you answered Marta about high availability in 07-07. When two options give the same result, the simpler one wins. And the test cluster stays, because a lab where you can practise without risk is worth far more than the decision not to use it in production.
In 08-06 the course closes. You are going to take everything built across eight modules into a going-to-production checklist of some thirty rows, each with its verifiable evidence — and that is where the encryption-in-transit debt that 08-01 resolved is formally closed. You will finally set up the monitoring with Prometheus and Grafana that was postponed in 05-07, with the four golden signals, the essential PromQL and metrics of your own: the age of the last backup, the days until the certificate expires, the replica's lag. You will configure alerts that are actionable, which is the opposite of what almost everybody does. You will write the daily operations procedure, the duty log and the blameless post-incident review. You will turn the 99.8 % proposed in 07-07 into an error budget that decides whether you deploy or stabilise. And you will close the three debts still open since Module 5: the off-site backup that is not append-only, the authorized_keys2 that nobody has explained, and the availability objective that was never formalised.
Linux Course: From Beginner to System Administrator
Module 1: Introduction to Linux
- What Is Linux?
- History of Linux
- Linux Distributions
- Installing Linux
- First Contact with the System
- The Linux File System Structure
Module 2: Basic Linux Commands
- Introduction to the Command Line
- Getting Help and System Documentation
- Navigating the File System
- File and Directory Operations
- Viewing and Editing Files
- Hard and Symbolic Links
- File Permissions and Ownership
Module 3: Advanced Command-Line Skills
- The Shell Environment: Variables, Aliases and History
- Using Wildcards and Regular Expressions
- Searching Files and Content: find, locate and grep
- Pipes and Redirection
- Text Processing: cut, sort, uniq, sed and awk
- Process Management
- Scheduling Tasks with Cron
- Networking Commands
Module 4: Shell Scripting
- Introduction to Shell Scripting
- Variables and Data Types
- Script Input, Output and Arguments
- Control Structures
- Functions and Libraries
- Debugging and Error Handling
- Production Scripts: Best Practices
Module 5: System Administration
- User and Group Management
- sudo and Special Permissions
- Package Management
- Disk Management
- systemd and Service Management
- System Logs: journald and syslog
- System Monitoring and Performance Tuning
- Backup and Restore
Module 6: Networking and Security
- Network Configuration
- SSH and Remote Access
- Firewalls and Perimeter Security
- Intrusion Detection Systems
- Secrets Management and TLS Certificates
- Securing Linux Systems
Module 7: Advanced Topics
- The Boot Process and System Recovery
- Advanced Diagnostics: strace, perf and eBPF
- Linux Kernel Tuning
- Virtualization with Linux
- Linux Containers and Docker
- Automation with Ansible
- High Availability and Load Balancing
