On AWS we obtained serious infrastructure with little operational effort: ECS Fargate runs the image, RDS keeps the data, the load balancer distributes traffic and nobody administers a server. The lesson ended with an open question: what happens when Ribalta's network stops being one application and becomes five, with three teams deploying on their own and the council asking whether this can be moved to another provider?
The industry's answer is Kubernetes, the container orchestrator that has become the de facto standard. This lesson tackles it honestly: first it explains what problem it solves and why, for a single application like CicloUrbana, it is usually unjustified complexity, and then it does the job properly, with the complete, annotated manifests for the Ribalta network. This is where two pieces we have been preparing since module 7 finally fall into place: the health probes from 07-01, which Kubernetes queries in three different ways, and the image from 07-04, which is exactly the unit a Deployment deploys.
Contents
- What problem an orchestrator solves, and when it pays off
- Kubernetes architecture and objects
- Practice environment and
kubectl Namespace,ConfigMapandSecret- CicloUrbana's
Deployment - The three probes and the health groups from 07-01
ServiceandIngresswith TLS- Flyway migrations:
JoborinitContainer - Rolling deployment,
rollout statusandrollout undo HPAandPodDisruptionBudget- Packaging with Helm
- PostgreSQL inside or outside the cluster
- Observability and debugging a pod that will not start
- GitOps and Spring Cloud Kubernetes
- Common Mistakes and Tips
- Exercises
- What problem an orchestrator solves, and when it pays off
An orchestrator takes a set of machines and presents them as a single computing resource to which you declare a desired state. Instead of saying "start this container on this machine", you say "I want three copies of this image, with this memory limit, reachable under this name", and the system takes care of achieving it and of maintaining it despite failures.
| Problem | How Kubernetes solves it |
|---|---|
| A container goes down | The controller detects it and creates another to return to the desired count |
| A machine dies | The pods are rescheduled onto the remaining ones |
| Spreading containers across machines | The scheduler places them according to resources, affinities and constraints |
| Service discovery | Every Service has a stable internal DNS name |
| Zero-downtime deployment and scaling | Progressive Deployment with rollback; HorizontalPodAutoscaler driven by metrics |
| Configuration and secrets | ConfigMap and Secret mounted as variables or files |
| Portability between providers | The same manifests on EKS, GKE, AKS or on your own server |
The honest warning, before we go on. Everything in that table was also done by ECS in 08-03, with a fraction of the effort. Kubernetes brings with it a vocabulary of dozens of objects, a control plane that has to be upgraded, add-ons to install (an ingress controller, cert-manager, metrics, a log aggregator), a non-trivial network model and a broad security surface. For a single application, ECS or a PaaS usually suffice and are usually better.
Kubernetes starts to pay off when several of these conditions coincide: there are many services (more than five or six) deploying independently; there are several teams that need autonomy without treading on each other, with Namespaces and quotas; portability between providers or self-hosting is required; advanced automation is needed —canary, GitOps, operators—; and there is already an internal platform with somebody maintaining it.
For CicloUrbana, the honest answer today is: it is not needed. We study it because it is the council's scenario three years out, because it is the destination of the image we already know how to build, and because the probes from 07-01 only reveal their full meaning here.
- Kubernetes architecture and objects
flowchart TD
subgraph CP["Control plane (managed by the provider)"]
API[API Server] --- ETCD[(etcd)]
API --- SCHED[Scheduler]
API --- CM[Controller Manager]
end
KUBECTL[kubectl / CI] --> API
subgraph N1["Node 1"]
K1[kubelet] --> P1[Pod ciclourbana]
K1 --> P2[Pod ciclourbana]
end
subgraph N2["Node 2"]
K2[kubelet] --> P3[Pod ciclourbana]
end
API --> K1
API --> K2
ING[Ingress Controller] --> SVC[Service ClusterIP]
SVC --> P1 & P2 & P3
The mental model is simple and worth fixing: you declare the desired state in the API Server, which stores it in etcd, and the controllers continuously compare the desired with the real and act to reduce the difference. You do not give orders: you describe a goal.
| Object | What it is | In CicloUrbana |
|---|---|---|
| Pod | The smallest unit: one or more containers sharing network and volumes | One pod = one instance of the application |
| ReplicaSet / Deployment | The Deployment manages ReplicaSets and orchestrates updates and rollbacks |
ciclourbana, 3 replicas |
| Service | A stable DNS name and virtual IP over a changing set of pods | ciclourbana:8080 inside the cluster |
| Ingress | HTTP/HTTPS routing from outside, with TLS and by host or path | ciclourbana.ribalta.example |
| ConfigMap | Non-sensitive configuration, as variables or files | Active profile, database URL |
| Secret | Sensitive data, base64-encoded | PostgreSQL password, JWT secret |
| Namespace | Logical partition of the cluster with its own names and quotas | ciclourbana-prod, ciclourbana-pre |
| HPA | Adjusts the number of replicas according to metrics | Scale by CPU between 3 and 10 |
| PVC | A request for persistent storage | We do not use it: the database is outside |
| Job / CronJob | A one-off or periodic process that runs to completion | The Flyway migrations |
- Practice environment and
kubectl
kubectlTo practise at no cost you do not need a paid cluster:
minikube (minikube start --cpus 4 --memory 6144) is the most complete, with ingress and metrics add-ons; kind (kind create cluster --name ribalta) is lightweight and extremely fast, ideal for CI itself; and Docker Desktop only asks you to tick the "Enable Kubernetes" box.
Cost warning: a managed cluster (EKS, GKE, AKS) costs around $70/month just for the control plane, plus the nodes. Practise locally; if you do create a managed one, destroy it the same day.
The essential commands:
| Command | What it does |
|---|---|
kubectl apply -f manifest.yaml |
Creates or updates what is declared (declarative mode) |
kubectl get pods -n ciclourbana-prod |
Lists pods and their status |
kubectl describe pod <name> |
Detail and events: the first port of call when debugging |
kubectl logs <pod> -f |
Follows the log; --previous for the container that died |
kubectl exec -it <pod> -- sh |
Opens a shell inside the container |
kubectl port-forward svc/ciclourbana 8080:8080 |
Local tunnel without exposing anything |
kubectl rollout status / undo deploy/ciclourbana |
Waits for the update to finish; rolls back to the previous one |
kubectl get events --sort-by=.lastTimestamp |
What has happened in the namespace |
One working rule: apply over files versioned in Git, never kubectl edit or kubectl run in production, because a change made by hand is lost at the next apply and nobody knows it ever existed.
Namespace, ConfigMap and Secret
Namespace, ConfigMap and SecretThe first manifest, 00-namespace.yaml, is trivial: a Namespace called ciclourbana-prod with the labels project: ciclourbana and environment: prod. It isolates names, permissions and quotas from the pre-production environment.
# 01-configmap.yaml — NON-sensitive configuration
apiVersion: v1
kind: ConfigMap
metadata:
name: ciclourbana-config
namespace: ciclourbana-prod
data:
SPRING_PROFILES_ACTIVE: "prod"
TZ: "Europe/Madrid"
JAVA_TOOL_OPTIONS: "-XX:MaxRAMPercentage=75 -XX:+ExitOnOutOfMemoryError"
SPRING_DATASOURCE_URL: "jdbc:postgresql://ciclourbana-prod.abc123xyz.eu-west-1.rds.amazonaws.com:5432/ciclourbana?sslmode=require"
SPRING_DATASOURCE_USERNAME: "ciclourbana_app"
SPRING_FLYWAY_ENABLED: "false"
MANAGEMENT_SERVER_PORT: "8081"
SERVER_FORWARD_HEADERS_STRATEGY: "framework"The keys are environment variable names, which Spring Boot translates into properties through relaxed binding (02-05): SPRING_DATASOURCE_URL → spring.datasource.url. Here we can keep the separate management port on 8081 from 07-01 —unlike on Heroku— because a pod can expose several ports, and the Service decides which ones it publishes.
# 02-secret.yaml — NEVER versioned with real values
apiVersion: v1
kind: Secret
metadata:
name: ciclourbana-secrets
namespace: ciclourbana-prod
type: Opaque
stringData: # stringData accepts plain text; K8s encodes it when storing
SPRING_DATASOURCE_PASSWORD: "REPLACE"
JWT_SECRET: "REPLACE"Fundamental warning: a Kubernetes Secret is only base64-encoded, not encrypted. Base64 is not encryption: kubectl get secret ciclourbana-secrets -o jsonpath='{.data.JWT_SECRET}' | base64 -d returns the value in clear for anybody with read permissions on the namespace. And by default it is stored as it is in etcd.
| Solution | What it provides | Cost |
|---|---|---|
| Strict RBAC | Only whoever needs it can read Secrets |
Free; essential in any case |
Encryption at rest for etcd |
The provider encrypts what is in etcd with KMS |
On EKS/GKE it is a cluster checkbox |
| Sealed Secrets | You version an encrypted SealedSecret; only the controller decrypts it |
One more controller; fits with GitOps |
| External Secrets Operator | Syncs Secrets from Secrets Manager, SSM or Vault |
One more operator; the best option with AWS (08-03) |
A Secret is never versioned with real values. You version the template with REPLACE, and the real value arrives through kubectl create secret --from-literal at initial creation, or —the correct way— through External Secrets from the store of 08-03.
- CicloUrbana's
Deployment
Deployment# 03-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: ciclourbana
namespace: ciclourbana-prod
labels: { app: ciclourbana }
spec:
replicas: 3
revisionHistoryLimit: 5
selector:
matchLabels: { app: ciclourbana }
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1 # at most 1 extra pod during the update
maxUnavailable: 0 # never below 3 healthy pods
template:
metadata:
labels: { app: ciclourbana, version: "2.4.0" }
spec:
terminationGracePeriodSeconds: 60
securityContext: { runAsNonRoot: true, runAsUser: 10001, fsGroup: 10001, seccompProfile: { type: RuntimeDefault } }
containers:
- name: ciclourbana
image: ghcr.io/ribalta-council/ciclourbana:2.4.0
imagePullPolicy: IfNotPresent
ports:
- { name: http, containerPort: 8080 }
- { name: management, containerPort: 8081 }
envFrom:
- configMapRef: { name: ciclourbana-config }
- secretRef: { name: ciclourbana-secrets }
resources:
requests: { cpu: "300m", memory: "768Mi" }
limits: { cpu: "1", memory: "1Gi" }
securityContext: { allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, capabilities: { drop: ["ALL"] } }
volumeMounts: [{ name: tmp, mountPath: /tmp }]
startupProbe: # up to 120 s to start
httpGet: { path: /actuator/health/liveness, port: management }
periodSeconds: 5
failureThreshold: 24
livenessProbe:
httpGet: { path: /actuator/health/liveness, port: management }
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 3
readinessProbe:
httpGet: { path: /actuator/health/readiness, port: management }
periodSeconds: 5
timeoutSeconds: 3
failureThreshold: 3
lifecycle: { preStop: { exec: { command: ["sh", "-c", "sleep 10"] } } }
volumes: [{ name: tmp, emptyDir: { sizeLimit: 256Mi } }]The points that really matter:
resources and its relationship with MaxRAMPercentage. requests is what the scheduler reserves in order to place the pod; limits is the hard ceiling. Exceeding the memory limit does not produce an exception: the kernel kills the process (OOMKilled). With limits.memory: 1Gi and MaxRAMPercentage=75 from the ConfigMap, the maximum heap is 768 MB and the remaining 256 MB cover metaspace, thread stacks, direct buffers and native code. Setting 100 % is the guaranteed recipe for an OOMKilled with no trace. On CPU, limits: 1 means one core: if you set a low limit, the kernel's throttling lengthens the JVM's startup and can make the startupProbe fail — which is why requests.cpu should be generous here.
securityContext at two levels. The pod's sets the unprivileged user (consistent with the non-root user of the 07-04 image); the container's forbids privilege escalation, drops every kernel capability and mounts the root filesystem read-only. That last one breaks the application if anything needs to write —Tomcat uses /tmp for uploads and internal work— which is why an emptyDir is mounted on /tmp: an ephemeral volume, specific to each pod, that disappears with it. It is factor 6 of 08-01 imposed by the system.
envFrom with configMapRef and secretRef injects all the keys of both objects at once as environment variables, so adding a property means editing the ConfigMap without touching the Deployment. The trade-off: changing a ConfigMap does not restart the pods, and the new configuration does not take effect until a kubectl rollout restart deploy/ciclourbana.
preStop and terminationGracePeriodSeconds, married to the graceful shutdown. The complete sequence when retiring a pod:
sequenceDiagram
participant K as Kubernetes
participant P as CicloUrbana pod
participant S as Service / Endpoints
K->>S: remove the pod from the endpoints
K->>P: run preStop (sleep 10)
Note over P: keeps serving requests<br/>while the Service propagates the change
K->>P: SIGTERM
Note over P: graceful shutdown (01-05):<br/>finishes in-flight work, closes executors and pool
Note over K: if still alive after 60 s -> SIGKILL
The sleep 10 in preStop is not a dirty trick: it is the standard solution to the fact that removing the endpoint and sending the SIGTERM happen in parallel, not in sequence. Without that margin, for a couple of seconds there is traffic directed at a pod that is already closing, and citizens see 502s. And the arithmetic has to add up: terminationGracePeriodSeconds (60) > preStop (10) + timeout-per-shutdown-phase (40), or Kubernetes will kill the process halfway through the graceful shutdown.
- The three probes and the health groups from 07-01
This is the section where 07-01 takes on its full meaning. Kubernetes asks three different questions and acts differently depending on the answer:
| Probe | Question | If it fails, Kubernetes... | Endpoint |
|---|---|---|---|
| startupProbe | Has it finished starting? | Keeps waiting; suspends the other two | /actuator/health/liveness |
| livenessProbe | Is the process unrecoverable? | Kills the container and restarts it | /actuator/health/liveness |
| readinessProbe | Can it serve requests right now? | Takes it out of the Service, without killing it |
/actuator/health/readiness |
And the configuration we already wrote in 07-01 makes those endpoints answer correctly:
management:
server:
port: 8081
endpoint:
health:
probes:
enabled: true
group:
readiness: { include: db } # includes the database
liveness: { include: livenessState } # does NOT include the databaseWhy confusing them causes restart loops. Suppose somebody puts the database in the liveness group, or points the livenessProbe at the full /actuator/health. RDS has a 40-second maintenance window:
All three pods stop reaching the database and /actuator/health returns 503; the livenessProbe fails three times in a row on all three; Kubernetes kills all three containers at once; they start again, the database is still under maintenance and they die again; and then the exponential backoff kicks in —CrashLoopBackOff, with waits of 10, 20, 40 seconds up to 5 minutes—. A 40-second incident becomes a twenty-minute one, and it is the orchestrator itself that draws it out.
With the correct configuration, what happens is very different: readiness fails, the pods leave the Service, the Ingress returns 503 for those 40 seconds —which is honest: the application genuinely cannot work— and liveness stays green because the process is perfectly healthy. When the database comes back, readiness goes UP and the pods receive traffic again. Without a single restart.
And the startupProbe solves the JVM's classic conflict. A Spring Boot application with Hibernate takes 25-45 seconds to start, and a livenessProbe with failureThreshold: 3 and periodSeconds: 10 would kill it after 30, before it managed to become alive. The old solution —a large initialDelaySeconds— delays failure detection for the pod's whole life; the startupProbe separates the two concerns: it allows up to 120 seconds to start (24 × 5 s) and once passed it hands control to the other two, which can then be aggressive and detect a real failure in 30 seconds.
Service and Ingress with TLS
Service and Ingress with TLS# 04-service.yaml
apiVersion: v1
kind: Service
metadata:
name: ciclourbana
namespace: ciclourbana-prod
spec:
type: ClusterIP # only reachable inside the cluster
selector: { app: ciclourbana }
ports:
- { name: http, port: 8080, targetPort: http }ClusterIP is deliberate: the Service publishes nothing to the outside, it only provides a stable internal DNS name (ciclourbana.ciclourbana-prod.svc.cluster.local) over whichever pods are ready at any given moment; what exposes it to the outside is the Ingress. And notice that the 8081 management port is not published: the probes query it directly on the pod, so Actuator is unreachable from outside the cluster by construction — the port separation of 07-01 doing its job.
# 05-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ciclourbana
namespace: ciclourbana-prod
annotations:
cert-manager.io/cluster-issuer: "letsencrypt-prod"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
ingressClassName: nginx
tls:
- { hosts: [ciclourbana.ribalta.example], secretName: ciclourbana-tls } # cert-manager creates and renews it
rules:
- host: ciclourbana.ribalta.example
http:
paths:
- path: /
pathType: Prefix
backend:
service: { name: ciclourbana, port: { name: http } }An Ingress is only a declaration: you need a controller to implement it (ingress-nginx, Traefik, or the provider's). The cert-manager.io/cluster-issuer annotation makes cert-manager —an operator you have to install— request the certificate from Let's Encrypt, validate the domain, store the certificate in the indicated Secret and renew it automatically before it expires. It is the equivalent of ACM in 08-03.
Since the controller terminates TLS and speaks HTTP to the pods, we still need what 08-01 described, and it is already in the ConfigMap: SERVER_FORWARD_HEADERS_STRATEGY: framework.
- Flyway migrations:
Job or initContainer
Job or initContainerThe ConfigMap disabled Flyway. The reason is the same as in 08-01 and 08-03, sharper here: with 3 replicas, three pods would start at once and try to migrate simultaneously. Flyway serialises it with a lock in the database, so nothing is corrupted, but the pods that wait consume their startup window, and if the migration fails all three go into CrashLoopBackOff and the application is down even though the previous version worked.
# 06-job-migration.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: ciclourbana-migration-2-4-0 # the name includes the version: one Job per deployment
namespace: ciclourbana-prod
spec:
backoffLimit: 2 # at most 3 attempts
ttlSecondsAfterFinished: 3600 # deletes itself after 1 h
template:
spec:
restartPolicy: Never # a Job does NOT restart: it fails or completes
containers:
- name: migration
image: ghcr.io/ribalta-council/ciclourbana:2.4.0 # THE SAME image
args:
- "--spring.flyway.enabled=true"
- "--spring.main.web-application-type=none"
envFrom:
- configMapRef: { name: ciclourbana-config }
- secretRef: { name: ciclourbana-secrets }
resources: { requests: { cpu: "200m", memory: "512Mi" }, limits: { cpu: "1", memory: "1Gi" } }A Job runs the pod until it finishes successfully. web-application-type=none brings up the context without Tomcat: Flyway migrates and the process exits. The deployment sequence is then:
kubectl apply -f 06-job-migration.yaml
kubectl wait --for=condition=complete --timeout=600s job/ciclourbana-migration-2-4-0 -n ciclourbana-prod
kubectl set image deploy/ciclourbana ciclourbana=ghcr.io/ribalta-council/ciclourbana:2.4.0 -n ciclourbana-prod
kubectl rollout status deploy/ciclourbana -n ciclourbana-prodIf the kubectl wait fails, the Deployment is not touched: the previous version carries on serving Ribalta perfectly normally.
The initContainer alternative runs the migration inside each pod before the main container. It is simpler to write, but N replicas run N migrations: with Flyway's lock nothing is corrupted, but the work is multiplied, every pod's startup is lengthened and you lose the Job's most valuable property —that a failed migration stops the deployment before touching the live pods—. Its only reasonable use case is a Deployment with a single replica.
And the usual reminder: during the rolling update 2.3.0 and 2.4.0 coexist against the already-migrated schema, so every migration must be backwards compatible — expand/contract from 04-08.
- Rolling deployment,
rollout status and rollout undo
rollout status and rollout undoWith maxSurge: 1 and maxUnavailable: 0 over 3 replicas, Kubernetes creates a fourth pod with the new version, waits for its readinessProbe to pass, adds it to the Service and only then retires an old one. And repeats. At no point are there fewer than 3 healthy pods serving.
The three usual settings: maxUnavailable: 0 with maxSurge: 1 guarantees capacity in exchange for a slower deployment and the need for headroom in the cluster; maxUnavailable: 1 with maxSurge: 1 is faster but allows a moment with only 2 pods; and maxSurge: 3 is the fastest, doubling consumption during the transition.
kubectl rollout status deploy/ciclourbana -n ciclourbana-prod # waits and fails if it does not progress
kubectl rollout history deploy/ciclourbana -n ciclourbana-prod # revisions
kubectl rollout undo deploy/ciclourbana [--to-revision=7] # roll back
kubectl rollout restart deploy/ciclourbana # restart without changing the imagerollout status is the piece the pipeline of 08-05 uses as its success criterion: it returns a non-zero code if the update does not progress within the deadline (progressDeadlineSeconds, 600 s by default), which makes automating the rollback possible.
And the warning from 08-01, which here is literal: rollout undo rolls back the application, not the database. The applied migration stays applied. If the schema is not backwards compatible, rolling back makes the situation worse instead of fixing it.
HPA and PodDisruptionBudget
HPA and PodDisruptionBudget# 07-hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: ciclourbana, namespace: ciclourbana-prod }
spec:
scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: ciclourbana }
minReplicas: 3
maxReplicas: 10
metrics:
- type: Resource
resource: { name: cpu, target: { type: Utilization, averageUtilization: 70 } }
behavior:
scaleUp: { stabilizationWindowSeconds: 30 }
scaleDown: { stabilizationWindowSeconds: 300 }The relationship between the HPA and resources is the key, and almost nobody sees it first time round: averageUtilization: 70 does not mean 70 % of a core, but 70 % of requests.cpu. With requests.cpu: 300m, the target is 210 millicores per pod. Two practical consequences follow: if requests is set very low, the HPA scales constantly for nothing; if it is set very high, it never scales even when the pods are struggling. Without requests.cpu, the HPA does not work at all and kubectl describe hpa shows <unknown> in the metrics column.
And as in 08-03: maxReplicas × maximum-pool-size + headroom < max_connections of PostgreSQL. With 10 replicas and a pool of 10 that is 100 connections, plus the migration Job and administration. You have to check it before raising maxReplicas, not when the traffic peak produces too many clients.
# 08-pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: ciclourbana, namespace: ciclourbana-prod }
spec:
minAvailable: 2
selector: { matchLabels: { app: ciclourbana } }The PodDisruptionBudget protects against voluntary disruptions: when an administrator drains a node (kubectl drain) or the node autoscaler consolidates machines, Kubernetes respects the budget and does not remove pods if that would leave fewer than 2 available. It does not protect against involuntary failures —a node that dies takes its pods with it anyway— but it eliminates the most frequent cause of avoidable outages in Kubernetes: a cluster upgrade that drains two nodes at once and takes all the replicas with it.
- Packaging with Helm
The eight manifests above describe one environment. For pre you need eight more almost identical ones, with a different namespace, replicas, database and domain: copy and paste is the guarantee that in three months' time pre and prod will diverge without anybody knowing how. Helm is Kubernetes' package manager: it turns the manifests into templates and separates the values by environment.
charts/ciclourbana/
├── Chart.yaml # name, chart version, appVersion
├── values.yaml # default values
├── values-pre.yaml # only what changes in pre
├── values-prod.yaml # only what changes in prod
└── templates/ # configmap, deployment, service, ingress,
│ # hpa, pdb, job-migration...
└── _helpers.tpl # name and label functions# values.yaml
replicaCount: 3
image: { repository: ghcr.io/ribalta-council/ciclourbana, tag: "2.4.0" }
resources: { requests: { cpu: 300m, memory: 768Mi }, limits: { cpu: "1", memory: 1Gi } }
ingress: { host: ciclourbana.ribalta.example }
config: { profile: prod, databaseUrl: "jdbc:postgresql://ciclourbana-prod...:5432/ciclourbana?sslmode=require" }
autoscaling: { enabled: true, min: 3, max: 10, targetCpu: 70 }# values-pre.yaml — ONLY the differences
replicaCount: 1
resources:
requests: { cpu: 100m, memory: 512Mi }
limits: { cpu: 500m, memory: 768Mi }
ingress: { host: pre.ciclourbana.ribalta.example }
config: { profile: pre, databaseUrl: "jdbc:postgresql://ciclourbana-pre...:5432/ciclourbana?sslmode=require" }
autoscaling: { enabled: false }And the template consumes those values:
# templates/deployment.yaml (fragment)
spec:
replicas: {{ .Values.replicaCount }}
template:
spec:
containers:
- name: ciclourbana
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
resources: {{- toYaml .Values.resources | nindent 12 }}helm lint ./charts/ciclourbana
helm template ciclourbana ./charts/ciclourbana -f values-pre.yaml # see the final YAML without applying
helm install ciclourbana ./charts/ciclourbana -n ciclourbana-pre -f values-pre.yaml --create-namespace
helm upgrade ciclourbana ./charts/ciclourbana -n ciclourbana-prod -f values-prod.yaml \
--set image.tag=2.4.1 --atomic --timeout 5m
helm rollback ciclourbana -n ciclourbana-prod # to the previous revision--atomic is the option that provides the most peace of mind: if the update does not complete successfully within the deadline, Helm automatically rolls back to the previous state. And --set image.tag=2.4.1 is the exact line the pipeline of 08-05 will run.
Alternative: Kustomize, built into kubectl (kubectl apply -k), which uses no templates but patches over a common base:
| Helm | Kustomize | |
|---|---|---|
| Mechanism | Go templates with values | Patches over base manifests |
| Learning curve and installation | Medium; a separate tool | Low (YAML over YAML); included in kubectl |
| Distributing to third parties | Yes: chart repositories | Not designed for it |
| Installation state | Yes, with history and rollback |
No: only apply |
Practical rule: Kustomize if only a few values change between pre and prod; Helm if the chart has logic or conditionals, or has to be distributed. For CicloUrbana, Helm pays off thanks to --atomic, rollback and the version history.
- PostgreSQL inside or outside the cluster
You can run PostgreSQL on Kubernetes with a StatefulSet and a PVC, or with a serious operator such as CloudNativePG or Zalando's. But:
| Managed (RDS, Cloud SQL) | Inside the cluster | |
|---|---|---|
| Backups, PITR and upgrades | Automatic and proven by the provider | You configure them, you test them |
| High availability | Multi-AZ with automatic failover | An operator you have to understand and operate |
| Storage performance | Optimised | Depends on the StorageClass and the network |
| Risk of data loss | Low | High if you do not master the operator |
| Cost | Higher on the bill | Lower on the bill, far higher in time |
The course recommends the managed database without qualification. Kubernetes is designed for stateless, disposable workloads —pods that get killed and recreated with no consequences— and a database is exactly the opposite. Modern operators are good, but the day a PVC gets corrupted or a network failure separates the primary from the replica, you need somebody who knows PostgreSQL and Kubernetes at the same time. That profile is scarce, and Ribalta council does not have it.
A reasonable exception: PostgreSQL inside the cluster in dev and in the tests, where losing the data costs nothing.
- Observability and debugging a pod that will not start
kubectl logs deploy/ciclourbana -n ciclourbana-prod -f --tail=100
kubectl logs <pod> --previous # the log of the container that DIED: key in CrashLoopBackOff
kubectl get events -n ciclourbana-prod --sort-by=.lastTimestampThe logs from kubectl logs come from the container's stdout and are lost when the pod disappears: in production you need an agent that forwards them to an aggregator (Fluent Bit, Vector or the provider's), which is the subject of 09-05.
The debugging method, in this exact order:
kubectl get pods -n ciclourbana-prod # 1. what state is it in?
kubectl describe pod <pod> -n ciclourbana-prod # 2. the Events section at the end: it is almost always there
kubectl logs <pod> --previous # 3. if it restarted, the previous container's log| State | Meaning | Usual causes |
|---|---|---|
Pending |
It could not be placed | No node with enough CPU/memory for the requests; unbound PVC |
ImagePullBackOff |
It cannot pull the image | Nonexistent tag, private registry with no imagePullSecrets, wrong name |
CreateContainerConfigError |
It cannot build the container | The referenced ConfigMap or Secret does not exist |
CrashLoopBackOff |
It starts and dies repeatedly | Exception during startup, failed migration, badly configured liveness |
Running but 0/1 READY |
Alive but not ready | readinessProbe failing: almost always it cannot reach the database |
OOMKilled |
The kernel killed it for memory | Low limits.memory or MaxRAMPercentage set too high |
OOMKilled in a JVM deserves a separate explanation, because it is misleading. It is not a Java OutOfMemoryError: there is no stack trace, no exception, nothing in the log. The process simply disappears and kubectl describe pod shows Last State: Terminated, Reason: OOMKilled, Exit Code: 137, because the JVM asked the system for more memory than the cgroup allows and the kernel killed it. The causes, by frequency: MaxRAMPercentage set too high or missing (the heap grows to the limit and the rest of the JVM's memory does not fit); limits.memory insufficient for what the application genuinely needs; and, last of all, a real leak, best diagnosed with -XX:+HeapDumpOnOutOfMemoryError and a volume to write the dump to.
And the practical distinction: a Java OutOfMemoryError does leave a stack trace in the log and, with ExitOnOutOfMemoryError (07-04), exits with code 1. An OOMKilled exits with 137 and leaves nothing. If you see 137, it is the kernel; if you see 1 with a trace, it is the heap.
To debug a pod with readOnlyRootFilesystem and no shell there is kubectl debug -it <pod> --image=busybox --target=ciclourbana, which attaches an ephemeral container with tools to the running pod.
- GitOps and Spring Cloud Kubernetes
GitOps reverses the direction of the deployment: instead of the pipeline pushing changes to the cluster with kubectl (which requires giving it administrator credentials), an agent inside the cluster watches a Git repository and applies what it finds.
The flow becomes: the developer opens a pull request against the manifest repository, the pipeline limits itself to updating image.tag in that repository, and Argo CD —which lives inside the cluster— detects the change, syncs it and continuously compares the real state with the declared one.
Concrete advantages: Git is the single source of truth and its history says who deployed what and when; a rollback is a git revert; the pipeline needs no cluster credentials, which greatly reduces the attack surface (08-05); and the agent detects drift, marking any hand-made change as OutOfSync. The reference tools are Argo CD (a very visual web interface) and Flux (lighter, integrated with Helm).
Spring Cloud Kubernetes, finally, lets the application read ConfigMaps and Secrets as property sources, discover services through the Kubernetes API and reload configuration at runtime. Often it is not needed, and it is important to know why: mounting the ConfigMap as variables (our envFrom) already solves configuration with zero dependencies; discovery is done by the internal DNS, which works with any HTTP client —the RestClient of 07-06 calls http://scooters:8080 and that is all—; and hot reloading adds complexity compared with a kubectl rollout restart, which is explicit, auditable and always works. It would also force us to give the application RBAC permissions over the API, widening its attack surface. It is worth it with dozens of services and centralised configuration; for CicloUrbana, the ConfigMap through envFrom is the right answer and it keeps the artefact agnostic of the platform, as 08-01 requires.
Common Mistakes and Tips
Putting the database in the liveness group. A 40-second maintenance window restarts every pod and causes CrashLoopBackOff for twenty minutes. Liveness = healthy process; readiness = can work.
Not setting a startupProbe. The livenessProbe kills the JVM before it finishes starting and the pod never becomes ready. And MaxRAMPercentage with no headroom produces an OOMKilled with code 137 and nothing in the log: leave 25 %.
Forgetting requests.cpu. The HPA cannot compute utilisation, shows <unknown> and never scales. And readOnlyRootFilesystem without mounting /tmp makes Tomcat fail when writing its working directory, with a far from obvious startup error.
Base64 secrets taken for encrypted ones. Base64 is undone with one command. Use strict RBAC, encryption at rest and External Secrets or Sealed Secrets.
Changing a ConfigMap and expecting it to apply on its own. The pods keep the variables they started with: you need kubectl rollout restart. And migrating from an initContainer with several replicas means N pods, N migrations, and a failure that brings down the whole deployment instead of stopping it beforehand.
Tip: kubectl describe pod before anything else. The Events section at the end explains 80 % of problems in plain text.
Tip: verify before applying. helm template shows the final resolved YAML and kubectl apply --dry-run=server -f validates it against the real API without creating anything: the pipeline's two verification steps.
Tip: label everything with app, version and environment. Selectors, log queries and metrics all depend on it.
Exercises
Exercise 1
Write CicloUrbana's Deployment for the pre environment: 1 replica, requests of 200m/512Mi and limits of 500m/768Mi, image 2.5.0-rc1, the three probes correctly configured and a hardened securityContext. Work out which MaxRAMPercentage value is appropriate and justify each failureThreshold, knowing that in pre the application takes about 50 seconds to start because it has less CPU.
Exercise 2
Version 2.4.0 is deployed and kubectl get pods shows, for ten minutes:
NAME READY STATUS RESTARTS AGE
ciclourbana-7d4b8c9f5-2xk9p 0/1 CrashLoopBackOff 6 9m
ciclourbana-7d4b8c9f5-8mq2w 0/1 CrashLoopBackOff 6 9m
ciclourbana-7d4b8c9f5-p4v7t 0/1 CrashLoopBackOff 6 9mDescribe the diagnostic procedure step by step and develop the five most likely causes of this particular symptom, with the evidence that confirms each one and its fix. Bear in mind that version 2.3.0 was working correctly until ten minutes ago.
Exercise 3
The council wants to deploy CicloUrbana to pre and prod from the same code, with one command per environment and without duplicating manifests. Design the Helm chart: what goes in values.yaml, what in values-pre.yaml and values-prod.yaml, how the migration Job is parameterised so that it runs once per version, and what the complete sequence of commands is that the pipeline of 08-05 would run to deploy version 2.4.1 to production safely and reversibly.
Solutions
Solution 1
Starting from the Deployment in section 5, what changes is the namespace to ciclourbana-pre, replicas: 1, the image to 2.5.0-rc1 and these blocks:
resources:
requests: { cpu: "200m", memory: "512Mi" }
limits: { cpu: "500m", memory: "768Mi" }
startupProbe:
httpGet: { path: /actuator/health/liveness, port: management }
periodSeconds: 5
failureThreshold: 30 # 150 s: three times the 50 s measured
livenessProbe:
httpGet: { path: /actuator/health/liveness, port: management }
periodSeconds: 10
failureThreshold: 3 # 30 s to detect an unrecoverable process
readinessProbe:
httpGet: { path: /actuator/health/readiness, port: management }
periodSeconds: 5
failureThreshold: 3 # 15 s to leave rotationThe hardened securityContext (runAsNonRoot, runAsUser: 10001, allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, capabilities: drop: ["ALL"]), the emptyDir on /tmp, the preStop and terminationGracePeriodSeconds: 60 stay identical: they do not depend on the environment.
MaxRAMPercentage. With limits.memory: 768Mi, 75 % is 576 MB of heap and 192 MB are left for everything else. In a JVM with Hibernate and Spring Security, metaspace (~90 MB), thread stacks (~40 MB) and direct buffers already come to 150-180 MB: that is far too tight. In small containers you have to be more conservative: MaxRAMPercentage=60 (460 MB of heap, 308 MB of headroom). It is a general and counter-intuitive principle: the smaller the container, the lower the percentage should be, because the JVM's fixed costs do not scale with size.
The failureThreshold values. The startupProbe is the only one that has to accommodate startup: 50 s measured, but with limits.cpu: 500m —half a core— the JVM suffers throttling and on a busy day it may take twice as long, so 30 attempts × 5 s = 150 s gives a threefold margin without being absurd. Once it has passed, liveness with 3 × 10 s detects an unrecoverable process in 30 seconds, and readiness with 3 × 5 s takes the pod out of rotation in 15. With replicas: 1, readiness failing means 503 for the whole of pre, which is correct and honest: there is nobody else to send the traffic to.
Solution 2
Procedure. First kubectl describe pod ciclourbana-7d4b8c9f5-2xk9p -n ciclourbana-prod and read the Events section and the Last State field (reason and exit code). Then kubectl logs ciclourbana-7d4b8c9f5-2xk9p --previous, which is the part most often forgotten: without --previous you ask for the current container's log, which has not written anything yet. And in parallel, kubectl get events --sort-by=.lastTimestamp and kubectl rollout history to see what changed with respect to 2.3.0.
The five causes, with their evidence:
1. The Flyway migration fails and ddl-auto: validate rejects the schema. The first suspicion, because 2.3.0 was working: what changed is the version and with it the expected schema. Evidence: in the previous log, a FlywayException or Schema-validation: missing column [total_docks] in table [stations]. Fix: check the Job (kubectl logs job/ciclourbana-migration-2-4-0); if it failed and the Deployment was updated anyway, the kubectl wait has been skipped — roll back only if the schema is still compatible and fix the pipeline so that the deployment depends on the Job.
2. OOMKilled. Evidence: Last State: Terminated, Reason: OOMKilled, Exit Code: 137, and the previous log shows no error at all: the application was starting normally and it disappears. Fix: raise limits.memory or lower MaxRAMPercentage; if 2.4.0 added a cache or a query that loads many objects, the limit has become too small.
3. A livenessProbe that is too aggressive or badly aimed. Evidence: in Events, Liveness probe failed: HTTP probe failed with statuscode: 503 followed by Killing container, and in the previous log the application reaches Started CicloUrbanaApplication and dies shortly after. Fix: verify that liveness includes only livenessState and not db, and that there is a startupProbe; if 2.4.0 starts more slowly than 2.3.0, the probe that used to be enough no longer is.
4. A Secret or ConfigMap that is missing or has changed. Evidence: the state would be CreateContainerConfigError if the object does not exist; if it exists with a wrong value, the previous log shows Could not resolve placeholder 'JWT_SECRET' or an authentication failure against PostgreSQL. Fix: kubectl get secret ciclourbana-secrets -o yaml and check that every key 2.4.0 needs is there — a new property in the code that was not added to the ConfigMap produces exactly this.
5. The image is not the one you think. Evidence: an Image ID that does not match, or exec format error in the previous log (an arm64 image built on Apple Silicon, 08-03). Fix: rebuild with --platform linux/amd64 and use immutable tags.
Golden rule for this scenario: all three pods fail the same way, so it is not a node problem nor an infrastructure one; it is the new version. And the correct immediate action —before investigating— is kubectl rollout undo deploy/ciclourbana, provided the schema is still backwards compatible. If cause 1 is the real one and the migration was destructive, rolling back fixes nothing: it is the scenario of exercise 3 in 08-01.
Solution 3
How the values are split. values.yaml holds everything common and everything that rarely changes: image repository, ports, the three probes with their thresholds, securityContext, preStop, terminationGracePeriodSeconds, labels and the Ingress annotations. values-pre.yaml and values-prod.yaml hold only the differences: replicaCount, resources, ingress.host, config.profile, config.databaseUrl, autoscaling and the name of the Secret. The criterion: if a value is the same in both environments, it cannot live in the per-environment files, or sooner or later they will diverge for no reason.
The Job parameterised by version. The key is for the name to include the image tag, so that each deployment creates a different Job (Kubernetes would refuse to recreate one with the same name) and a trace of every migration remains:
# templates/job-migration.yaml (fragment)
metadata:
name: {{ include "ciclourbana.fullname" . }}-migration-{{ .Values.image.tag | replace "." "-" }}
annotations:
"helm.sh/hook": pre-upgrade,pre-install
"helm.sh/hook-weight": "-5"
"helm.sh/hook-delete-policy": before-hook-creation
spec:
backoffLimit: 2
template:
spec:
restartPolicy: Never
containers:
- name: migration
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
args: ["--spring.flyway.enabled=true", "--spring.main.web-application-type=none"]Helm hooks are the elegant piece: pre-upgrade makes Helm run the Job and wait for it to finish successfully before touching the Deployment. If it fails, helm upgrade aborts and the pods carry on with the previous version — exactly the property we were after in section 8, now with no intermediate scripts.
The pipeline's sequence for 2.4.1:
# 1. Static verification, without touching the cluster
helm lint ./charts/ciclourbana
helm template ciclourbana ./charts/ciclourbana -f values-prod.yaml --set image.tag=2.4.1 | kubectl apply --dry-run=server -f -
# 2. Deploy to pre and wait for it to converge
helm upgrade --install ciclourbana ./charts/ciclourbana -n ciclourbana-pre -f values-pre.yaml --set image.tag=2.4.1 --atomic --timeout 8m
# 3. Smoke test against pre
curl -fsS https://pre.ciclourbana.ribalta.example/actuator/health/readiness
curl -fsS https://pre.ciclourbana.ribalta.example/api/v1/stations | jq 'length'
# 4. Manual approval (a protected GitHub Actions environment, 08-05)
# 5. Production: the hook migrates, and if it fails the Deployment is not touched
helm upgrade ciclourbana ./charts/ciclourbana -n ciclourbana-prod -f values-prod.yaml --set image.tag=2.4.1 --atomic --timeout 10m
# 6. Verify that what is running is what you think, and roll back if not
kubectl rollout status deploy/ciclourbana -n ciclourbana-prod
curl -fsS https://ciclourbana.ribalta.example/actuator/info | jq '.build.version'
helm rollback ciclourbana -n ciclourbana-prodWhy it is safe and reversible. --atomic rolls back on its own if the deployment does not converge within the deadline; the pre-upgrade hook guarantees that a failed migration stops the process before touching the live pods; helm rollback returns to the previous revision in one command; the /actuator/info with build-info from 07-01 confirms which version is really running; and the underlying condition, the usual one: the rollback only works if 2.4.1's migration is backwards compatible (expand/contract, 04-08).
Conclusion
CicloUrbana now runs on an orchestrator, and it does so knowing why and when that makes sense. The lesson began with a warning worth not forgetting: for a single application, ECS or a PaaS usually suffice and are usually better, and Kubernetes only pays off when there are many services, several autonomous teams, a portability requirement or advanced automation. With that honesty as the starting point, you have built Ribalta's whole network on the cluster.
You know the architecture —control plane, etcd, scheduler, controllers and kubelet— and the mental model that explains it: you declare the desired state and the controllers reduce the difference with reality. You handle the objects that matter and you have the annotated manifests: a Namespace per environment, a ConfigMap with the non-sensitive configuration and a Secret with the clear warning that base64 is not encryption, together with the real solutions —strict RBAC, encryption at rest, Sealed Secrets and External Secrets Operator, the best option when the store is already on AWS (08-03)—.
The Deployment brings together everything the course had been preparing: the 07-04 image, requests and limits married to MaxRAMPercentage —with the principle that the smaller the container, the lower the percentage should be—, variables from configMapRef and secretRef, a securityContext with a non-root user, dropped capabilities and a read-only filesystem with /tmp on an emptyDir, and the ten-second preStop with a generous terminationGracePeriodSeconds that makes the graceful shutdown of 01-05 fit together with the propagation of the endpoints. And above all, the three probes: startupProbe so that the JVM has time to start without liveness killing it, livenessProbe over livenessState to restart only what is unrecoverable, and readinessProbe over the group that includes db to leave rotation without dying. You know exactly how a 40-second database maintenance window turns into twenty minutes of CrashLoopBackOff if they are confused, and why the configuration from 07-01 prevents it.
You also have the ClusterIP Service that does not publish the management port, the Ingress with TLS renewed by cert-manager, the migrations in a Job that stops the deployment if it fails, the rolling update with maxUnavailable: 0 and its rollout undo with the warning about the database, the HPA with its relationship to requests.cpu that almost nobody sees first time round, the PodDisruptionBudget, and the Helm chart with per-environment values, hooks for the migration and --atomic. You know when Kustomize is better, why the database should stay managed, how to debug a pod that will not start —describe first, Events at the end, logs --previous next— and what CrashLoopBackOff, ImagePullBackOff and an OOMKilled with code 137 that leaves no trace in the log actually mean.
And here the tour of the platforms ends. CicloUrbana can be deployed on a PaaS, on managed containers or on an orchestrator, and in all three cases the underlying decisions are those of 08-01. But in all three lessons there has been an implicit actor who was still human: somebody who runs git push heroku main, aws ecs update-service or helm upgrade from their laptop, with their credentials, hoping they tested beforehand what they are deploying. The module's last lesson, Continuous Integration and Delivery, removes that actor: a pipeline that compiles, runs the module 6 tests —Testcontainers included—, analyses quality, builds and scans the image, publishes it tagged with the commit SHA, deploys to pre, passes the smoke tests and, after an explicit approval, takes it to production without anybody touching a server by hand.
Spring Boot Course
Module 1: Introduction to Spring Boot
- What Is Spring Boot?
- Setting Up Your Development Environment
- Building Your First Spring Boot Application
- Understanding the Project Structure
- Application Startup and Lifecycle
Module 2: Spring Boot Core Concepts
- Spring Boot Annotations
- Dependency Injection in Spring Boot
- Bean Scope and Lifecycle
- Spring Boot Configuration
- Spring Boot Properties
- Auto-Configuration and Starters from the Inside
Module 3: Building RESTful Web Services
- Introduction to RESTful Web Services
- Creating REST Controllers
- Handling HTTP Methods
- Validating Input Data
- DTOs and Mapping Between Layers
- Exception Handling in REST
- Documenting the API with OpenAPI
Module 4: Data Access with Spring Boot
- Introduction to Spring Data JPA
- Configuring Data Sources
- Creating JPA Entities
- Relationships Between Entities
- Using Spring Data Repositories
- Query Methods in Spring Data JPA
- Transactions and Persistence Management
- Schema Migrations with Flyway
Module 5: Security in Spring Boot
- Introduction to Spring Security
- Configuring Spring Security
- User Authentication and Authorization
- Implementing JWT Authentication
- Method-Level Security and API Hardening
Module 6: Testing in Spring Boot
- Introduction to Testing
- Unit Testing with JUnit
- Mocking with Mockito
- Integration Testing
- Testing with Testcontainers
Module 7: Advanced Spring Boot Features
- Spring Boot Actuator
- Spring Boot Profiles
- Scheduled Tasks and Asynchronous Execution
- Spring Boot with Docker
- Spring Boot and Microservices
- Service Communication and Fault Tolerance
Module 8: Deploying Spring Boot Applications
- Introduction to Deployment
- Deploying to Heroku
- Deploying to AWS
- Deploying to Kubernetes
- Continuous Integration and Delivery
Module 9: Performance and Monitoring
- Performance Tuning
- Caching with Spring Cache
- Monitoring with Spring Boot Actuator
- Using Prometheus and Grafana
- Logging and Log Management
- Distributed Tracing
