In the previous lesson we changed the image of web-store and observed something we deliberately left unexplained: a second ReplicaSet appeared, the old one dropped to zero replicas without being deleted, and for a few seconds pods from two versions coexisted. That mechanism is the Deployment's whole reason for being and the direct answer to the first of Rutas Norte's four goals: zero downtime on deployments, against the minute or two of outage that today forces deployments into the small hours of Tuesday. In this lesson we take control of that transfer. You will see the Deployment's two native strategies and when each one is required, you will calculate pod by pod what happens during an update of bookings-api, you will tune the pace with maxSurge and maxUnavailable, you will tell apart which changes generate a new revision and which do not, and you will handle the full kubectl rollout toolkit: status, history, undo, pause and resume. And you will finish by breaking a rollout on purpose with a non-existent image, diagnosing it and undoing it.
Contents
- The two native strategies:
RollingUpdateandRecreate - Why
bookings-postgrescannot takeRollingUpdate maxSurgeandmaxUnavailable, step by step withbookings-api- Pace and tolerance:
minReadySeconds,progressDeadlineSeconds,revisionHistoryLimit - What triggers a new revision and what does not
- The full
kubectl rollouttoolkit - The
kubernetes.io/change-causeannotation - Zero-downtime deployment: what it really takes
- Diagnosing and rolling back a stalled deployment
- The two native strategies:
RollingUpdate and Recreate
RollingUpdate and Recreatespec.strategy.type accepts only two values. There are no other native strategies in Kubernetes: blue-green and canary, which you will see in 11-04, are built by combining these bricks with Services and Ingress.
RollingUpdate (default)
It replaces the pods gradually: it brings up pods of the new version and retires the old ones little by little, always keeping a minimum number of pods serving.
flowchart LR
subgraph T0["Start"]
A1["v1"]; A2["v1"]; A3["v1"]; A4["v1"]
end
subgraph T1["Halfway"]
B1["v1"]; B2["v1"]; B3["v2"]; B4["v2"]
end
subgraph T2["End"]
C1["v2"]; C2["v2"]; C3["v2"]; C4["v2"]
end
T0 --> T1 --> T2
Recreate
It kills every old pod, waits for them to disappear, and only then creates the new ones. There is an interval, usually from seconds to a couple of minutes, with no pod serving at all.
flowchart LR
subgraph R0["Start"]
D1["v1"]; D2["v1"]
end
subgraph R1["SERVICE OUTAGE"]
X["No pods"]
end
subgraph R2["End"]
E1["v2"]; E2["v2"]
end
R0 --> R1 --> R2
Comparison
| Aspect | RollingUpdate |
Recreate |
|---|---|---|
| Service outage | No | Yes, for as long as the replacement takes |
| Two versions coexist | Yes, unavoidably | No, never |
| Extra capacity needed | Yes, if maxSurge > 0 |
No |
| Speed | Slower (gradual) | Faster |
| Peak resource usage | Up to replicas + maxSurge |
Never above replicas |
| Rollback | Gradual, also without downtime | With another outage |
| When to use it | Stateless workloads: web-store, bookings-api, notifications-worker |
When two versions cannot coexist |
How to declare it explicitly:
- Why
bookings-postgres cannot take RollingUpdate
bookings-postgres cannot take RollingUpdateThis is the key question of this section, and the answer teaches more than the strategy itself. Imagine bookings-postgres were a 1-replica Deployment with RollingUpdate and maxSurge: 1. During the update, this is what would happen:
- Kubernetes creates the new pod before retiring the old one, because
maxSurge: 1allows it. - For a few seconds there are two simultaneous PostgreSQL processes.
- Both try to mount and write to the same data directory.
Real consequences, in order of severity:
- Startup lock-out: PostgreSQL detects the live instance's
postmaster.pidfile and refuses to start. The rollout stalls (the benign case). - Data corruption if the lock fails or the storage does not guarantee it: two processes writing to the same WAL files.
- Lost writes: bookings confirmed to the customer that are never recorded.
And there is a second reason, independent of the storage: schema migration. If the new version of bookings-api requires a column the old one knows nothing about, having both versions talking to the same database at once breaks one of them. With RollingUpdate that coexistence lasts minutes; with Recreate, it does not exist.
| Component | Strategy | Reason |
|---|---|---|
web-store |
RollingUpdate |
Stateless; two versions of an SPA coexist without trouble |
bookings-api |
RollingUpdate |
Stateless; the API contract stays compatible between consecutive versions |
notifications-worker |
RollingUpdate |
Stateless; it consumes from a queue, two versions can consume at once |
redis-cache |
Recreate |
A single replica with a volume; and the cache repopulates itself anyway |
bookings-postgres |
Neither: StatefulSet | It needs stable identity and storage (06-01) |
The honest conclusion: for bookings-postgres, the right answer is not Recreate, it is not to use a Deployment. Recreate is the appropriate strategy for single-replica workloads that cannot tolerate duplication but do not need a stable identity either, like our redis-cache.
Apply it to redis-cache, which you already deployed in the previous lesson:
kubectl patch deployment redis-cache --type=merge \
-p '{"spec":{"strategy":{"type":"Recreate","rollingUpdate":null}}}'
kubectl get deploy redis-cache -o jsonpath='{.spec.strategy.type}{"\n"}'Remember to carry that change into the manifest in Git as well.
maxSurge and maxUnavailable, step by step with bookings-api
maxSurge and maxUnavailable, step by step with bookings-apiThese two parameters completely govern the pace of a rolling update.
| Parameter | What it limits | Default value | Effect of raising it |
|---|---|---|---|
maxSurge |
How many extra pods there can be above replicas |
25 % | Faster update, more resources at the peak |
maxUnavailable |
How many pods can be unavailable below replicas |
25 % | Faster update, less capacity during the process |
Both accept an integer or a percentage. Percentages are rounded like this: maxSurge up and maxUnavailable down, so as to always err on the side of extra capacity. And there is one restriction: they cannot both be 0 at the same time, because then the rollout could not take a single step.
The scenario
bookings-api with 4 replicas, maxSurge: 1 and maxUnavailable: 1. We update from version 2.4.0 to 2.5.0. The numbers that govern the whole process:
- Maximum simultaneous pods =
replicas + maxSurge= 4 + 1 = 5 - Minimum available pods =
replicas - maxUnavailable= 4 − 1 = 3
Set up the scenario:
kubectl scale deployment bookings-api --replicas=4
kubectl patch deployment bookings-api --type=merge \
-p '{"spec":{"strategy":{"type":"RollingUpdate","rollingUpdate":{"maxSurge":1,"maxUnavailable":1}}}}'
kubectl get deploy bookings-apiThe exact count, step by step
| Step | Controller action | v2.4.0 pods | v2.5.0 pods | Total | Available | Within the limits? |
|---|---|---|---|---|---|---|
| 0 | Initial state | 4 ready | 0 | 4 | 4 | Yes |
| 1 | Creates 1 new pod (uses the maxSurge) |
4 ready | 1 being created | 5 | 4 | Total = 5 = maximum |
| 2 | Retires 1 old pod (uses the maxUnavailable) |
3 ready | 1 being created | 4 | 3 | Available = 3 = minimum |
| 3 | The new pod becomes Ready |
3 ready | 1 ready | 4 | 4 | Yes |
| 4 | Creates another new pod | 3 ready | 1 ready + 1 being created | 5 | 4 | Total = 5 |
| 5 | Retires another old one | 2 ready | 1 ready + 1 being created | 4 | 3 | Available = 3 |
| 6 | The second new one becomes Ready |
2 ready | 2 ready | 4 | 4 | Yes |
| 7 | Creates the third | 2 ready | 2 ready + 1 being created | 5 | 4 | Total = 5 |
| 8 | Retires the third old one | 1 ready | 2 ready + 1 being created | 4 | 3 | Available = 3 |
| 9 | The third becomes Ready |
1 ready | 3 ready | 4 | 4 | Yes |
| 10 | Creates the fourth | 1 ready | 3 ready + 1 being created | 5 | 4 | Total = 5 |
| 11 | Retires the last old one | 0 | 3 ready + 1 being created | 4 | 3 | Available = 3 |
| 12 | The fourth becomes Ready. Done |
0 | 4 ready | 4 | 4 | Complete |
The two important readings of that table:
- At no point are there fewer than 3 pods serving. Rutas Norte never loses total capacity, only 25 % for a few seconds.
- At no point are there more than 5 pods. The cluster needs room for one extra pod, not for eight.
flowchart TD
P0["Step 0<br/>4 old · 0 new<br/>total 4"] --> P1["Step 1<br/>4 old · 1 new<br/>total 5 ← maxSurge ceiling"]
P1 --> P2["Step 2<br/>3 old · 1 new<br/>available 3 ← maxUnavailable floor"]
P2 --> P3["Step 3<br/>the new one becomes Ready<br/>available 4"]
P3 --> PN["The cycle repeats<br/>create · retire · wait for Ready"]
PN --> PF["Step 12<br/>0 old · 4 new<br/>rollout complete"]
Watching it live
In one terminal:
In another, launch the update:
NAME READY STATUS RESTARTS AGE
bookings-api-6b4c9d7f5-x2jkp 1/1 Running 0 41m
bookings-api-6b4c9d7f5-k7wpd 1/1 Running 0 41m
bookings-api-6b4c9d7f5-m4rzt 1/1 Running 0 41m
bookings-api-6b4c9d7f5-q8vnc 1/1 Running 0 41m
bookings-api-9f2a7c531-t3xkb 0/1 Pending 0 0s
bookings-api-9f2a7c531-t3xkb 0/1 ContainerCreating 0 0s
bookings-api-6b4c9d7f5-x2jkp 1/1 Terminating 0 41m
bookings-api-9f2a7c531-t3xkb 1/1 Running 0 3s
bookings-api-9f2a7c531-w9djm 0/1 ContainerCreating 0 0s
bookings-api-6b4c9d7f5-k7wpd 1/1 Terminating 0 41m
...And the replica count of each ReplicaSet during the process:
The Deployment is shifting replicas from one ReplicaSet to the other. That is, literally, all a rolling update does.
Common combinations
maxSurge |
maxUnavailable |
Behaviour | When to use it |
|---|---|---|---|
25% |
25% |
The default: a balance between speed and safety | Most cases |
1 |
0 |
Never drops below nominal capacity. Bring up first, retire second | Production with critical capacity: bookings-api in rutas-norte-pro |
0 |
1 |
Never exceeds nominal capacity. Retire first, bring up second | Clusters with very tight resources or narrow quotas |
100% |
0 |
Brings up all the new ones at once and then retires all the old ones | A very fast rollout, requires double the resources |
0 |
0 |
Invalid: the API rejects it | — |
For Rutas Norte production, the recommended choice is maxSurge: 1 and maxUnavailable: 0: on a bank-holiday weekend with traffic multiplied by six, losing 25 % of capacity during a rollout is not acceptable.
- Pace and tolerance:
minReadySeconds, progressDeadlineSeconds, revisionHistoryLimit
minReadySeconds, progressDeadlineSeconds, revisionHistoryLimitThree fields that complete the control over a rollout.
| Field | Default | What it does | Why it matters |
|---|---|---|---|
minReadySeconds |
0 |
Seconds a pod must stay ready without crashing before counting as available | Stops the rollout advancing over pods that start and die after 5 seconds |
progressDeadlineSeconds |
600 |
Seconds without progress after which the rollout is declared failed | Turns a silent stall into a detectable Progressing: False condition |
revisionHistoryLimit |
10 |
How many old ReplicaSets (at 0 replicas) are kept | It is your rollback capability; at 0, you can undo nothing |
minReadySeconds deserves a separate explanation because its effect is subtle but very valuable. With the default value of 0, as soon as a new pod says "I am ready", the controller retires another old one and carries on. If that pod crashes two seconds later, the damage is done: the rollout is advancing over a broken version. With minReadySeconds: 15, each new pod has to survive 15 seconds ready before counting. If it crashes sooner, the rollout stops.
progressDeadlineSeconds is the difference between a rollout that fails silently and one that raises the alarm. Without it, a rollout stalled by a non-existent image would keep trying forever without any alerting system noticing.
The bookings-api manifest with the three fields and the production strategy:
# k8s/base/bookings-api-deployment.yaml (updated fragment)
spec:
replicas: 4
revisionHistoryLimit: 5
minReadySeconds: 15
progressDeadlineSeconds: 300
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0With this configuration, an update of bookings-api in rutas-norte-pro takes at least 4 × 15 = 60 seconds, and if something fails it is declared failed after 5 minutes instead of 10. Slowness in exchange for safety: on a component that handles ticket payments, that is an excellent trade.
- What triggers a new revision and what does not
This is the most important rule in the lesson and the most forgotten:
Only changes under
spec.templatetrigger a new revision. Nothing else does.
The logic is transparent: a revision is a ReplicaSet, and one ReplicaSet is told apart from another by the hash of its pod template. If the template does not change, the hash does not change, and there is no new ReplicaSet to create.
| Change | Creates a revision? | What happens |
|---|---|---|
spec.template.spec.containers[].image |
Yes | New ReplicaSet; gradual replacement of pods |
spec.template.spec.containers[].env |
Yes | Same: the pods are recreated with the new variables |
spec.template.spec.containers[].resources |
Yes | Same |
spec.template.metadata.labels or annotations |
Yes | Same, even if the container is identical |
spec.template.spec.terminationGracePeriodSeconds |
Yes | Same |
spec.replicas |
No | The current ReplicaSet raises or lowers its number of pods |
spec.strategy |
No | It will apply on the next update |
spec.minReadySeconds |
No | Same |
spec.revisionHistoryLimit |
No | It only cleans up old ReplicaSets |
The Deployment's metadata.annotations |
No | Including change-cause: it is Deployment metadata, not pod metadata |
Check it for yourself:
kubectl rollout history deployment/bookings-api | tail -3
kubectl scale deployment bookings-api --replicas=5
kubectl rollout history deployment/bookings-api | tail -3REVISION CHANGE-CAUSE
1 Initial rollout of bookings-api 2.4.0
2 <none>
deployment.apps/bookings-api scaled
REVISION CHANGE-CAUSE
1 Initial rollout of bookings-api 2.4.0
2 <none>Same history: scaling is not deploying.
A practical consequence that surprises many people: changing a ConfigMap does not restart the pods that consume it, because the ConfigMap is not in the template, it is only referenced from it. The usual trick, which you will see in module 3, is to force a change in the template with a computed annotation, or to use directly:
rollout restart changes nothing functional: it writes a kubectl.kubernetes.io/restartedAt annotation inside the template, which alters the hash and triggers a gradual replacement of every pod. It is the correct, downtime-free way to "restart" a Deployment.
- The full
kubectl rollout toolkit
kubectl rollout toolkitkubectl set image: update the image
The syntax is <container-name>=<image>. The container name is the one in spec.template.spec.containers[].name, not the pod's. With several containers you can update them all at once by separating them with spaces.
A word on discipline: in a declarative project, the correct route is to edit the manifest and apply it. set image is for emergencies, and its change is lost on the next apply.
kubectl rollout status: wait
Waiting for deployment "bookings-api" rollout to finish: 1 out of 4 new replicas have been updated...
Waiting for deployment "bookings-api" rollout to finish: 2 out of 4 new replicas have been updated...
Waiting for deployment "bookings-api" rollout to finish: 3 out of 4 new replicas have been updated...
Waiting for deployment "bookings-api" rollout to finish: 1 old replicas are pending termination...
deployment "bookings-api" successfully rolled outkubectl rollout history: the history
deployment.apps/bookings-api
REVISION CHANGE-CAUSE
1 Initial rollout of bookings-api 2.4.0
2 Update to node 20.15 for a security patch
3 Update to node 20.16And the full detail of one particular revision:
deployment.apps/bookings-api with revision #2
Pod Template:
Labels: app=bookings-api
app.kubernetes.io/part-of=rutas-norte
environment=dev
pod-template-hash=9f2a7c531
Annotations: kubernetes.io/change-cause: Update to node 20.15 for a security patch
Containers:
api:
Image: node:20.15-alpine
Port: 3000/TCP
Limits: cpu: 500m, memory: 256Mi
Requests: cpu: 100m, memory: 128MiThat information comes entirely from the ReplicaSet at 0 replicas that was kept. That is why revisionHistoryLimit: 0 would leave you with no history and no rollback.
kubectl rollout undo: undo
It goes back to the immediately previous revision. To go to a specific one:
Two important clarifications:
- A rollback is just another deployment: it uses the same strategy, respects
maxSurgeandmaxUnavailable, and therefore does not interrupt the service either. - Undoing does not delete the bad revision: it creates a new revision with the old one's content. If you were on 3 and you
undo, you end up on 4, whose content is that of 2. The numbers never go backwards.
REVISION CHANGE-CAUSE
1 Initial rollout of bookings-api 2.4.0
3 Update to node 20.16
4 Update to node 20.15 for a security patchNotice that revision 2 has disappeared from the list: its ReplicaSet has been reused as revision 4.
kubectl rollout pause and resume: freeze midway
With the Deployment paused, the changes you make are not applied. It is the tool for grouping several modifications into a single rollout:
kubectl rollout pause deployment/bookings-api
kubectl set image deployment/bookings-api api=node:20.17-alpine
kubectl set resources deployment/bookings-api -c=api --limits=cpu=600m,memory=384Mi
kubectl scale deployment bookings-api --replicas=5
# so far NOTHING has happened in the cluster
kubectl rollout resume deployment/bookings-api
kubectl rollout status deployment/bookings-apideployment.apps/bookings-api paused
deployment.apps/bookings-api image updated
deployment.apps/bookings-api resource requirements updated
deployment.apps/bookings-api scaled
deployment.apps/bookings-api resumed
Waiting for deployment "bookings-api" rollout to finish: 3 of 5 updated replicas are available...
deployment "bookings-api" successfully rolled outWithout pause, those three commands would have caused three chained rollouts, each one interrupting the previous. With pause, a single one and a single revision.
The other use of pause is for emergencies: if you see a rollout producing broken pods, kubectl rollout pause freezes it on the spot, leaving whatever old pods remain serving while you decide whether to fix or undo.
Summary table of the full toolkit:
| Command | What for |
|---|---|
rollout status |
Wait and verify; returns a non-zero code on failure |
rollout history |
See revisions; with --revision=N, the detail |
rollout undo |
Go back, with or without --to-revision |
rollout pause / resume |
Freeze and resume; group changes |
rollout restart |
Recreate every pod without changing anything |
- The
kubernetes.io/change-cause annotation
kubernetes.io/change-cause annotationYou will have noticed the <none> entries in the CHANGE-CAUSE column. A revision history without reasons is nearly useless: three weeks later nobody remembers what revision 7 was.
kubernetes.io/change-cause is an annotation on the Deployment whose value is copied to that revision's ReplicaSet and shown in the history. Three ways to fill it in, from worst to best:
With kubectl annotate (quick, imperative):
kubectl annotate deployment/bookings-api \
kubernetes.io/change-cause="Update to node 20.17 and raised limits for the August peaks" \
--overwriteIn the manifest (the correct one according to the project conventions):
metadata:
name: bookings-api
annotations:
kubernetes.io/change-cause: "v2.5.0 - availability cache per departure (RN-482)"Automated in CI/CD, which is ideal:
kubectl annotate deployment/bookings-api \
kubernetes.io/change-cause="${CI_COMMIT_SHA} · ${CI_COMMIT_MESSAGE} · ${CI_USER}" \
--overwriteA detail you already know from section 5, but worth repeating because it causes confusion: changing only this annotation does not trigger a new revision, because it lives in the Deployment's metadata and not in the template. Annotate it together with the real change, not afterwards.
Good practice for the text at Rutas Norte: image version, a plain-language summary of the change and the ticket reference. A real example from the project:
REVISION CHANGE-CAUSE
1 v2.4.0 - initial rollout
2 v2.4.1 - fix to the free-seat calculation (RN-455)
3 v2.5.0 - availability cache per departure (RN-482)
4 v2.4.1 - ROLLBACK: 2.5.0 returned 500 when booking (RN-489)
- Zero-downtime deployment: what it really takes
Here comes the most important warning of the lesson. You might think that with a well-configured RollingUpdate you already have zero-downtime deployments. You do not, and this is the number one cause of 502 errors during supposedly perfect rollouts.
The problem is that, with what we know so far, Kubernetes considers a pod "ready" as soon as the process starts. But a Node.js process takes a few seconds to load dependencies, connect to PostgreSQL and become operational. During that window:
- The pod already counts as
Ready. - The Deployment confidently retires an old pod.
- The Service starts sending it real traffic.
- The new pod still cannot respond: 502 errors.
flowchart TD
A["A new pod starts"] --> B["Kubernetes marks it Ready<br/>merely because the process is alive"]
B --> C["The Service sends it traffic"]
C --> D{"Is the application<br/>really operational?"}
D -->|Yes| E["All correct"]
D -->|No: still connecting to PostgreSQL| F["502s to real customers<br/>for several seconds"]
The missing piece is called a readiness probe (readinessProbe): a check the kubelet runs against your application that decides whether the pod is genuinely ready to receive traffic. With it, step 1 of the diagram does not happen until the application truly responds.
The complete checklist for a genuinely downtime-free rollout at Rutas Norte:
| Requirement | Status right now | Where it is solved |
|---|---|---|
A well-parameterised RollingUpdate strategy |
Done in this lesson | 02-04 |
| More than one replica | Done | 02-03 |
minReadySeconds as a cushion |
Done | 02-04 |
A readinessProbe reflecting real availability |
Pending | 07-01 |
A livenessProbe to detect hung pods |
Pending | 07-01 |
Graceful shutdown that catches SIGTERM |
Done | 02-01 |
| A Service routing to the ready pods | Pending | 02-05 |
| Compatibility between consecutive API versions | Development's responsibility | — |
In other words: RollingUpdate is necessary but not sufficient. Without probes, the rolling update advances blind. Until we get to module 7, minReadySeconds is a poor but useful substitute.
- Diagnosing and rolling back a stalled deployment
The final exercise and the most realistic of all. One Friday afternoon, somebody deploys a version to rutas-norte-dev that does not exist in the registry.
The disaster
kubectl annotate deployment/bookings-api \
kubernetes.io/change-cause="v2.6.0 - version that does not exist (RN-501)" --overwrite
kubectl set image deployment/bookings-api api=node:99.99-nonexistent
kubectl rollout status deployment/bookings-api --timeout=90sdeployment.apps/bookings-api annotated
deployment.apps/bookings-api image updated
Waiting for deployment "bookings-api" rollout to finish: 1 out of 5 new replicas have been updated...
error: timed out waiting for the conditionStep 1: the overall state
Reading: there are 5 ready and available pods —the service is still running on the previous version— but only 1 has the new version. The rollout took one step and stopped. This is the system working properly: maxUnavailable: 0 prevented any old pod being retired until the new one was ready, and since it never will be, none has been retired.
Step 2: the ReplicaSets
The new ReplicaSet has 1 pod that never reaches READY.
Step 3: the guilty pod
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulling 2m kubelet Pulling image "node:99.99-nonexistent"
Warning Failed 2m kubelet Failed to pull image: manifest for node:99.99-nonexistent not found
Warning Failed 2m kubelet Error: ErrImagePull
Normal BackOff 30s (x6 over 2m) kubelet Back-off pulling image "node:99.99-nonexistent"Root cause identified, using exactly the procedure from the Pods lesson.
Step 4: the conditions
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing False ProgressDeadlineExceededAvailable: True (we are still serving) + Progressing: False with ProgressDeadlineExceeded (the rollout is dead) is the exact signature of a stalled deployment. In a monitored cluster, that combination should raise an alert.
Step 5: decide and act
Two options, depending on the moment:
# Option A: freeze while investigating (does not revert, only stops)
kubectl rollout pause deployment/bookings-api
# Option B: undo right now (the normal move on a Friday afternoon)
kubectl rollout undo deployment/bookings-api
kubectl rollout status deployment/bookings-apiAn important warning: if the Deployment is paused, undo has no effect. You have to resume first.
Step 6: verify and document
kubectl get deploy,rs -l app=bookings-api
kubectl annotate deployment/bookings-api \
kubernetes.io/change-cause="ROLLBACK to v2.5.0: the v2.6.0 image does not exist in the registry (RN-501)" \
--overwrite
kubectl rollout history deployment/bookings-apiNAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/bookings-api 5/5 5 5 1h
NAME DESIRED CURRENT READY AGE
replicaset.apps/bookings-api-3d8e1f602 0 0 0 6m
replicaset.apps/bookings-api-c5a9b47d8 5 5 5 22m
REVISION CHANGE-CAUSE
1 Initial rollout of bookings-api 2.4.0
4 v2.6.0 - version that does not exist (RN-501)
5 ROLLBACK to v2.5.0: the v2.6.0 image does not exist in the registry (RN-501)The broken ReplicaSet stays at 0 replicas, consuming nothing, as a record of the incident. And the history tells the whole story to whoever reads it in six months' time.
The moral of the whole section: during those six steps, bookings-api did not miss a single booking. With the original Docker Compose setup, that same mistake would have left the platform down until somebody noticed. That is what a well-configured Deployment buys you.
Leave the cluster tidy before moving on:
NAME READY UP-TO-DATE AVAILABLE AGE
bookings-api 2/2 2 2 1h
redis-cache 1/1 1 1 50m
web-store 3/3 3 3 1h
notifications-worker 2/2 2 2 45mCommon Mistakes and Tips
- Believing that
RollingUpdateguarantees zero downtime on its own. Without areadinessProbe, traffic reaches pods that cannot respond yet. It is the number one cause of 502s during rollouts. - Setting
maxSurge: 0andmaxUnavailable: 0. The API rejects it: the rollout could not advance. - Using
RollingUpdatewith a workload that cannot tolerate two simultaneous versions. Single-replica databases, processes with an exclusive lock or incompatible schema migrations: those call forRecreate, or a StatefulSet outright. - Setting
revisionHistoryLimit: 0to "save resources". ReplicaSets at 0 replicas consume neither CPU nor memory, just a few kilobytes in etcd. In exchange, you lose the ability to undo entirely. - Expecting a change to
spec.strategyorspec.replicasto trigger a rollout. They are not in thetemplate. They only apply on the next real update. - Expecting a change to a ConfigMap to restart the pods. It does not. Use
kubectl rollout restart. - Forgetting
change-cause. A history full of<none>is no use at all when you have to decide which revision to go back to at three in the morning. - Annotating the
change-causeafter the rollout. Since it does not trigger a revision, it stays attached to the Deployment but may not reflect what that revision did. Annotate it in the sameapply. - Running
undoon a paused Deployment. Nothing happens until youresume. It is a classic of blind debugging. - Trusting
kubectl applyas verification.applyonly says the API accepted the object. Always chainkubectl rollout status --timeout=...in your scripts. - Tip: in production,
maxSurge: 1andmaxUnavailable: 0. It costs one extra pod of capacity and guarantees you never drop below nominal capacity. - Tip:
kubectl rollout statusreturns a non-zero exit code if the rollout fails. That is what makes a CI/CD pipeline trustworthy:kubectl apply -f k8s/ && kubectl rollout status deploy/bookings-api --timeout=300s || kubectl rollout undo deploy/bookings-api.
Exercises
Exercise 1: Work out a rollout on paper
web-store is in rutas-norte-pro with 6 replicas, maxSurge: 2 and maxUnavailable: 1.
- What is the maximum number of simultaneous pods and the minimum number of available pods?
- Build a table with the first five steps of the update, showing for each one: old pods, new pods, total and available.
- If the platform team demands that the number of serving pods never drops below 6, what values would you set and what extra resources does the cluster need?
- With the configuration from point 3 and
minReadySeconds: 20, what is the theoretical minimum duration of the rollout if each pod takes 5 seconds to start?
Exercise 2: A complete update and rollback cycle
On web-store in rutas-norte-dev:
- Configure
RollingUpdatewithmaxSurge: 1,maxUnavailable: 0,minReadySeconds: 10andrevisionHistoryLimit: 5, applying it from the manifest. - Update to
nginx:1.27.1-alpinewith its correspondingchange-causeand wait for it to finish. - Update to
nginx:1.26-alpinewith anotherchange-cause. - Show the history and the detail of revision 2.
- Pretend 1.26 has a serious bug: go back to 1.27.1 and verify that all three replicas serve it.
- Explain why the history does not show the numbering you would expect.
Exercise 3: Diagnose a stalled rollout of notifications-worker
- Break
notifications-workeron purpose by deploying the imagebusybox:9.9.9-nonexistent, with achange-causeincluded. - With four commands, diagnose the problem following the correct order: Deployment state, ReplicaSets, guilty pod and conditions.
- Answer: are the confirmation emails still being sent during the stall? Justify it with the commands' output.
- Freeze the rollout, check that
undodoes nothing while it is paused, resume it and undo. - Document the rollback with an annotation and show the final history.
Solutions
Solution 1
-
Maximum pods =
replicas + maxSurge= 6 + 2 = 8. Minimum available =replicas - maxUnavailable= 6 − 1 = 5. -
The first five steps:
| Step | Action | Old | New | Total | Available |
|---|---|---|---|---|---|
| 0 | Initial state | 6 ready | 0 | 6 | 6 |
| 1 | Creates 2 new pods (maxSurge ceiling) |
6 ready | 2 being created | 8 | 6 |
| 2 | Retires 1 old one (maxUnavailable ceiling) |
5 ready | 2 being created | 7 | 5 |
| 3 | The 2 new ones become Ready |
5 ready | 2 ready | 7 | 7 |
| 4 | Creates 1 more new one (7 + 1 = 8, the ceiling) | 5 ready | 2 ready + 1 being created | 8 | 7 |
| 5 | Retires 2 old ones (available 7 − 2 = 5, the floor) | 3 ready | 2 ready + 1 being created | 6 | 5 |
-
To never drop below 6 serving:
maxUnavailable: 0, andmaxSurgeat 1 or 2 depending on how fast you want to go. WithmaxSurge: 2, the cluster must have room for 8web-storepods at once: withrequestsof 50m of CPU and 64Mi per pod, that is an extra 100m of CPU and 128Mi of memory free during the rollout. WithmaxUnavailable: 0the cluster must have that capacity; if it does not, the new pods stayPendingand the rollout never advances. -
With
maxSurge: 2andmaxUnavailable: 0, pods are replaced in batches of 2. Each batch takes 5 s (startup) + 20 s (minReadySeconds) = 25 s. Six pods in batches of two make 3 batches: 75 seconds as a theoretical minimum, not counting image pulls or the old pods' termination time.
Solution 2
# k8s/base/web-store-deployment.yaml (fragment)
spec:
replicas: 3
revisionHistoryLimit: 5
minReadySeconds: 10
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0kubectl apply -f k8s/base/web-store-deployment.yaml
# 2. Update to 1.27.1
kubectl annotate deployment/web-store \
kubernetes.io/change-cause="v1.27.1 - nginx security patch (RN-470)" --overwrite
kubectl set image deployment/web-store nginx=nginx:1.27.1-alpine
kubectl rollout status deployment/web-store
# 3. Update to 1.26
kubectl annotate deployment/web-store \
kubernetes.io/change-cause="v1.26 - back to the previous branch for compatibility (RN-473)" --overwrite
kubectl set image deployment/web-store nginx=nginx:1.26-alpine
kubectl rollout status deployment/web-store# 4. History and detail
kubectl rollout history deployment/web-store
kubectl rollout history deployment/web-store --revision=2deployment.apps/web-store
REVISION CHANGE-CAUSE
1 Initial rollout of web-store 1.27.0
2 v1.27.1 - nginx security patch (RN-470)
3 v1.26 - back to the previous branch for compatibility (RN-473)
deployment.apps/web-store with revision #2
Pod Template:
Labels: app=web-store
pod-template-hash=6f9c4b8d7
Annotations: kubernetes.io/change-cause: v1.27.1 - nginx security patch (RN-470)
Containers:
nginx:
Image: nginx:1.27.1-alpine# 5. Rollback to revision 2
kubectl rollout undo deployment/web-store --to-revision=2
kubectl rollout status deployment/web-store
kubectl get pods -l app=web-store \
-o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].imagedeployment.apps/web-store rolled back
deployment "web-store" successfully rolled out
NAME IMAGE
web-store-6f9c4b8d7-42kxr nginx:1.27.1-alpine
web-store-6f9c4b8d7-8vnwq nginx:1.27.1-alpine
web-store-6f9c4b8d7-t9mzd nginx:1.27.1-alpineREVISION CHANGE-CAUSE
1 Initial rollout of web-store 1.27.0
3 v1.26 - back to the previous branch for compatibility (RN-473)
4 v1.27.1 - nginx security patch (RN-470)- The numbering is surprising because revisions never go backwards. When you undo to revision 2, Kubernetes does not "return" to it: it reuses its ReplicaSet and renumbers it as revision 4, the most recent. That is why 2 disappears from the list and a 4 appears with the same
change-cause. The revision number identifies the order in which the changes were applied, not the content: a rollback is just another change.
Solution 3
# 1. Break it
kubectl annotate deployment/notifications-worker \
kubernetes.io/change-cause="v1.3.0 - image that does not exist (RN-512)" --overwrite
kubectl set image deployment/notifications-worker worker=busybox:9.9.9-nonexistent# 2. Diagnosis in four steps
kubectl get deploy notifications-worker
kubectl get rs -l app=notifications-worker
kubectl get pods -l app=notifications-worker
kubectl describe pod <pod-in-ImagePullBackOff> | tail -6
kubectl describe deployment notifications-worker | grep -A4 ConditionsNAME READY UP-TO-DATE AVAILABLE AGE
notifications-worker 2/2 1 2 1h
NAME DESIRED CURRENT READY AGE
notifications-worker-5a3f9d21c 1 1 0 90s
notifications-worker-8c7b5d94f 2 2 2 1h
NAME READY STATUS RESTARTS AGE
notifications-worker-5a3f9d21c-r6bkt 0/1 ImagePullBackOff 0 90s
notifications-worker-8c7b5d94f-k2xrt 1/1 Running 0 1h
notifications-worker-8c7b5d94f-p9mzq 1/1 Running 0 1h
Warning Failed 80s kubelet Failed to pull image "busybox:9.9.9-nonexistent": manifest unknown
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing False ProgressDeadlineExceeded- Yes, the emails are still going out. The proof is in three places:
READY 2/2andAVAILABLE 2on the Deployment, the 2 pods of the previous revision inRunning, and above allAvailable: TruewithMinimumReplicasAvailablein the conditions. OnlyUP-TO-DATE 1gives away that a new version is trying to get in and failing. Direct confirmation:
[pod/notifications-worker-8c7b5d94f-k2xrt/worker] sending batch of confirmations from notifications-worker-8c7b5d94f-k2xrt# 4. Pause, check that undo does nothing, resume and undo
kubectl rollout pause deployment/notifications-worker
kubectl rollout undo deployment/notifications-worker
kubectl get rs -l app=notifications-workerdeployment.apps/notifications-worker paused
deployment.apps/notifications-worker rolled back
NAME DESIRED CURRENT READY AGE
notifications-worker-5a3f9d21c 1 1 0 4m
notifications-worker-8c7b5d94f 2 2 2 1hThe broken ReplicaSet still has 1 replica: the undo has been recorded but not executed, because the Deployment is paused.
kubectl rollout resume deployment/notifications-worker
kubectl rollout status deployment/notifications-worker
kubectl get rs -l app=notifications-workerdeployment.apps/notifications-worker resumed
deployment "notifications-worker" successfully rolled out
NAME DESIRED CURRENT READY AGE
notifications-worker-5a3f9d21c 0 0 0 5m
notifications-worker-8c7b5d94f 2 2 2 1h# 5. Document
kubectl annotate deployment/notifications-worker \
kubernetes.io/change-cause="ROLLBACK to v1.2.0: the v1.3.0 image does not exist in the registry (RN-512)" \
--overwrite
kubectl rollout history deployment/notifications-workerREVISION CHANGE-CAUSE
1 Initial rollout of notifications-worker 1.2.0
2 v1.3.0 - image that does not exist (RN-512)
3 ROLLBACK to v1.2.0: the v1.3.0 image does not exist in the registry (RN-512)Conclusion
You have closed the circle we opened in the previous lesson with that second ReplicaSet that appeared without explanation. You now know that the Deployment's two native strategies are RollingUpdate and Recreate, and why the choice is not a matter of taste but of the workload's nature: web-store, bookings-api and notifications-worker tolerate two versions coexisting and go with RollingUpdate; redis-cache does not, and goes with Recreate; and bookings-postgres takes neither, because a Deployment is not where it belongs: two PostgreSQL processes over the same data directory mean a lock-out in the best case and corruption in the worst.
You can work out a rollout on paper before running it: replicas + maxSurge is the ceiling of pods and replicas - maxUnavailable the floor of available ones, and with those two numbers you can reconstruct step by step what the controller will do with the 4 replicas of bookings-api. You know the useful combinations and the one recommended for production, maxSurge: 1 with maxUnavailable: 0, which protects nominal capacity on an August peak. You tune the pace with minReadySeconds, you detect stalls with progressDeadlineSeconds and you preserve your ability to undo with revisionHistoryLimit. You are clear on the rule that governs everything: only changes under spec.template create a revision, which is why scaling does not deploy, changing a ConfigMap restarts nothing and kubectl rollout restart exists. And you handle the full toolkit —status, history, undo, pause, resume— with change-cause documenting every step, including the rollback of a deployment stalled by a non-existent image, resolved without a single customer failing to buy their ticket.
One warning and one big gap remain. The warning: RollingUpdate is necessary but not sufficient for genuinely downtime-free deployments; without a readinessProbe, Kubernetes marks a pod whose process has just started as ready and sends it traffic before it can serve it. That piece arrives in Health Checks and Probes. And the gap: all the traffic we have spent two lessons talking about does not exist yet. Our pods have IPs that change on every rollout, and to talk to bookings-api we have had to look up a specific pod's IP by hand. In the next lesson, Services, we will give each Rutas Norte component a stable address that survives rollouts, load balancing across replicas included.
Kubernetes Course
Module 1: Introduction to Kubernetes
- What Is Kubernetes?
- Kubernetes Architecture
- Key Concepts and Terminology
- Setting Up a Kubernetes Cluster
- The Kubernetes CLI: kubectl
- Objects, YAML Manifests and the Declarative Model
- The Course Project: the Rutas Norte Platform
Module 2: Core Kubernetes Components
- Pods
- ReplicaSets
- Deployments
- Updates, Rollbacks and Deployment Strategies
- Services
- Namespaces
- Labels, Selectors and Annotations
Module 3: Configuration and Secret Management
- ConfigMaps
- Secrets
- Environment Variables
- Resource Quotas and Limits
- LimitRanges and Quality of Service (QoS) Classes
- ServiceAccounts and API Access from Pods
Module 4: Networking in Kubernetes
- Cluster Networking
- Service Types
- Internal DNS and Service Discovery
- Ingress Controllers
- TLS and Certificate Management with cert-manager
- Network Policies
Module 5: Storage in Kubernetes
- Volumes
- Persistent Volumes
- Persistent Volume Claims
- Storage Classes
- Dynamic Provisioning, Expansion and Snapshots
- Backup and Restore of Persistent Data
Module 6: Advanced Kubernetes Concepts
- StatefulSets
- DaemonSets
- Jobs and CronJobs
- Init Containers, Sidecars and Multi-Container Patterns
- Scheduling: Affinity, Taints and Tolerations
- Custom Resource Definitions (CRDs)
- Operators and the Controller Pattern
Module 7: Monitoring and Logging
- Health Checks and Probes
- Metrics Server and kubectl top
- Monitoring with Prometheus
- Visualization and Alerting with Grafana and Alertmanager
- Centralized Logging with Elasticsearch, Fluentd and Kibana (EFK)
- Application Debugging and Cluster Events
Module 8: Kubernetes Security
- Role-Based Access Control (RBAC)
- Security Contexts and Container Hardening
- Pod Security Policies and Pod Security Standards
- Network Security
- Image Security
- Auditing, Scanning and Vulnerability Management
Module 9: Scaling and Performance
- Horizontal Pod Autoscaling
- Vertical Pod Autoscaling
- Cluster Autoscaling
- Event-Driven and Custom-Metric Scaling with KEDA
- High Availability: PodDisruptionBudgets and Topology
- Performance Tuning
Module 10: Kubernetes Ecosystem and Tooling
- Minikube and Local Environments with kind
- Kubeadm
- Helm
- Kustomize
- GitOps with Argo CD and Flux
- Managed Kubernetes: EKS, AKS and GKE
Module 11: Case Studies and Real-World Applications
- Deploying a Web Application
- Running Stateful Applications
- CI/CD with Kubernetes
- Deployment Strategies: Blue-Green and Canary
- Multi-Cluster Management
- Production Operations: Incidents, Runbooks and Costs
