In the previous lesson we changed the image of web-store and observed something we deliberately left unexplained: a second ReplicaSet appeared, the old one dropped to zero replicas without being deleted, and for a few seconds pods from two versions coexisted. That mechanism is the Deployment's whole reason for being and the direct answer to the first of Rutas Norte's four goals: zero downtime on deployments, against the minute or two of outage that today forces deployments into the small hours of Tuesday. In this lesson we take control of that transfer. You will see the Deployment's two native strategies and when each one is required, you will calculate pod by pod what happens during an update of bookings-api, you will tune the pace with maxSurge and maxUnavailable, you will tell apart which changes generate a new revision and which do not, and you will handle the full kubectl rollout toolkit: status, history, undo, pause and resume. And you will finish by breaking a rollout on purpose with a non-existent image, diagnosing it and undoing it.

Contents

  1. The two native strategies: RollingUpdate and Recreate
  2. Why bookings-postgres cannot take RollingUpdate
  3. maxSurge and maxUnavailable, step by step with bookings-api
  4. Pace and tolerance: minReadySeconds, progressDeadlineSeconds, revisionHistoryLimit
  5. What triggers a new revision and what does not
  6. The full kubectl rollout toolkit
  7. The kubernetes.io/change-cause annotation
  8. Zero-downtime deployment: what it really takes
  9. Diagnosing and rolling back a stalled deployment

  1. The two native strategies: RollingUpdate and Recreate

spec.strategy.type accepts only two values. There are no other native strategies in Kubernetes: blue-green and canary, which you will see in 11-04, are built by combining these bricks with Services and Ingress.

RollingUpdate (default)

It replaces the pods gradually: it brings up pods of the new version and retires the old ones little by little, always keeping a minimum number of pods serving.

flowchart LR
    subgraph T0["Start"]
        A1["v1"]; A2["v1"]; A3["v1"]; A4["v1"]
    end
    subgraph T1["Halfway"]
        B1["v1"]; B2["v1"]; B3["v2"]; B4["v2"]
    end
    subgraph T2["End"]
        C1["v2"]; C2["v2"]; C3["v2"]; C4["v2"]
    end
    T0 --> T1 --> T2

Recreate

It kills every old pod, waits for them to disappear, and only then creates the new ones. There is an interval, usually from seconds to a couple of minutes, with no pod serving at all.

flowchart LR
    subgraph R0["Start"]
        D1["v1"]; D2["v1"]
    end
    subgraph R1["SERVICE OUTAGE"]
        X["No pods"]
    end
    subgraph R2["End"]
        E1["v2"]; E2["v2"]
    end
    R0 --> R1 --> R2

Comparison

Aspect RollingUpdate Recreate
Service outage No Yes, for as long as the replacement takes
Two versions coexist Yes, unavoidably No, never
Extra capacity needed Yes, if maxSurge > 0 No
Speed Slower (gradual) Faster
Peak resource usage Up to replicas + maxSurge Never above replicas
Rollback Gradual, also without downtime With another outage
When to use it Stateless workloads: web-store, bookings-api, notifications-worker When two versions cannot coexist

How to declare it explicitly:

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
spec:
  strategy:
    type: Recreate      # no sub-fields: Recreate takes neither maxSurge nor maxUnavailable

  1. Why bookings-postgres cannot take RollingUpdate

This is the key question of this section, and the answer teaches more than the strategy itself. Imagine bookings-postgres were a 1-replica Deployment with RollingUpdate and maxSurge: 1. During the update, this is what would happen:

  1. Kubernetes creates the new pod before retiring the old one, because maxSurge: 1 allows it.
  2. For a few seconds there are two simultaneous PostgreSQL processes.
  3. Both try to mount and write to the same data directory.

Real consequences, in order of severity:

  • Startup lock-out: PostgreSQL detects the live instance's postmaster.pid file and refuses to start. The rollout stalls (the benign case).
  • Data corruption if the lock fails or the storage does not guarantee it: two processes writing to the same WAL files.
  • Lost writes: bookings confirmed to the customer that are never recorded.

And there is a second reason, independent of the storage: schema migration. If the new version of bookings-api requires a column the old one knows nothing about, having both versions talking to the same database at once breaks one of them. With RollingUpdate that coexistence lasts minutes; with Recreate, it does not exist.

Component Strategy Reason
web-store RollingUpdate Stateless; two versions of an SPA coexist without trouble
bookings-api RollingUpdate Stateless; the API contract stays compatible between consecutive versions
notifications-worker RollingUpdate Stateless; it consumes from a queue, two versions can consume at once
redis-cache Recreate A single replica with a volume; and the cache repopulates itself anyway
bookings-postgres Neither: StatefulSet It needs stable identity and storage (06-01)

The honest conclusion: for bookings-postgres, the right answer is not Recreate, it is not to use a Deployment. Recreate is the appropriate strategy for single-replica workloads that cannot tolerate duplication but do not need a stable identity either, like our redis-cache.

Apply it to redis-cache, which you already deployed in the previous lesson:

kubectl patch deployment redis-cache --type=merge \
  -p '{"spec":{"strategy":{"type":"Recreate","rollingUpdate":null}}}'
kubectl get deploy redis-cache -o jsonpath='{.spec.strategy.type}{"\n"}'
deployment.apps/redis-cache patched
Recreate

Remember to carry that change into the manifest in Git as well.

  1. maxSurge and maxUnavailable, step by step with bookings-api

These two parameters completely govern the pace of a rolling update.

Parameter What it limits Default value Effect of raising it
maxSurge How many extra pods there can be above replicas 25 % Faster update, more resources at the peak
maxUnavailable How many pods can be unavailable below replicas 25 % Faster update, less capacity during the process

Both accept an integer or a percentage. Percentages are rounded like this: maxSurge up and maxUnavailable down, so as to always err on the side of extra capacity. And there is one restriction: they cannot both be 0 at the same time, because then the rollout could not take a single step.

The scenario

bookings-api with 4 replicas, maxSurge: 1 and maxUnavailable: 1. We update from version 2.4.0 to 2.5.0. The numbers that govern the whole process:

  • Maximum simultaneous pods = replicas + maxSurge = 4 + 1 = 5
  • Minimum available pods = replicas - maxUnavailable = 4 − 1 = 3

Set up the scenario:

kubectl scale deployment bookings-api --replicas=4
kubectl patch deployment bookings-api --type=merge \
  -p '{"spec":{"strategy":{"type":"RollingUpdate","rollingUpdate":{"maxSurge":1,"maxUnavailable":1}}}}'
kubectl get deploy bookings-api
NAME           READY   UP-TO-DATE   AVAILABLE   AGE
bookings-api   4/4     4            4           41m

The exact count, step by step

Step Controller action v2.4.0 pods v2.5.0 pods Total Available Within the limits?
0 Initial state 4 ready 0 4 4 Yes
1 Creates 1 new pod (uses the maxSurge) 4 ready 1 being created 5 4 Total = 5 = maximum
2 Retires 1 old pod (uses the maxUnavailable) 3 ready 1 being created 4 3 Available = 3 = minimum
3 The new pod becomes Ready 3 ready 1 ready 4 4 Yes
4 Creates another new pod 3 ready 1 ready + 1 being created 5 4 Total = 5
5 Retires another old one 2 ready 1 ready + 1 being created 4 3 Available = 3
6 The second new one becomes Ready 2 ready 2 ready 4 4 Yes
7 Creates the third 2 ready 2 ready + 1 being created 5 4 Total = 5
8 Retires the third old one 1 ready 2 ready + 1 being created 4 3 Available = 3
9 The third becomes Ready 1 ready 3 ready 4 4 Yes
10 Creates the fourth 1 ready 3 ready + 1 being created 5 4 Total = 5
11 Retires the last old one 0 3 ready + 1 being created 4 3 Available = 3
12 The fourth becomes Ready. Done 0 4 ready 4 4 Complete

The two important readings of that table:

  • At no point are there fewer than 3 pods serving. Rutas Norte never loses total capacity, only 25 % for a few seconds.
  • At no point are there more than 5 pods. The cluster needs room for one extra pod, not for eight.
flowchart TD
    P0["Step 0<br/>4 old · 0 new<br/>total 4"] --> P1["Step 1<br/>4 old · 1 new<br/>total 5 ← maxSurge ceiling"]
    P1 --> P2["Step 2<br/>3 old · 1 new<br/>available 3 ← maxUnavailable floor"]
    P2 --> P3["Step 3<br/>the new one becomes Ready<br/>available 4"]
    P3 --> PN["The cycle repeats<br/>create · retire · wait for Ready"]
    PN --> PF["Step 12<br/>0 old · 4 new<br/>rollout complete"]

Watching it live

In one terminal:

kubectl get pods -l app=bookings-api -w

In another, launch the update:

kubectl set image deployment/bookings-api api=node:20.15-alpine
NAME                     READY   STATUS              RESTARTS   AGE
bookings-api-6b4c9d7f5-x2jkp   1/1   Running             0        41m
bookings-api-6b4c9d7f5-k7wpd   1/1   Running             0        41m
bookings-api-6b4c9d7f5-m4rzt   1/1   Running             0        41m
bookings-api-6b4c9d7f5-q8vnc   1/1   Running             0        41m
bookings-api-9f2a7c531-t3xkb   0/1   Pending             0        0s
bookings-api-9f2a7c531-t3xkb   0/1   ContainerCreating   0        0s
bookings-api-6b4c9d7f5-x2jkp   1/1   Terminating         0        41m
bookings-api-9f2a7c531-t3xkb   1/1   Running             0        3s
bookings-api-9f2a7c531-w9djm   0/1   ContainerCreating   0        0s
bookings-api-6b4c9d7f5-k7wpd   1/1   Terminating         0        41m
...

And the replica count of each ReplicaSet during the process:

kubectl get rs -l app=bookings-api
NAME                     DESIRED   CURRENT   READY   AGE
bookings-api-6b4c9d7f5   2         2         2       42m
bookings-api-9f2a7c531   3         3         2       25s

The Deployment is shifting replicas from one ReplicaSet to the other. That is, literally, all a rolling update does.

Common combinations

maxSurge maxUnavailable Behaviour When to use it
25% 25% The default: a balance between speed and safety Most cases
1 0 Never drops below nominal capacity. Bring up first, retire second Production with critical capacity: bookings-api in rutas-norte-pro
0 1 Never exceeds nominal capacity. Retire first, bring up second Clusters with very tight resources or narrow quotas
100% 0 Brings up all the new ones at once and then retires all the old ones A very fast rollout, requires double the resources
0 0 Invalid: the API rejects it

For Rutas Norte production, the recommended choice is maxSurge: 1 and maxUnavailable: 0: on a bank-holiday weekend with traffic multiplied by six, losing 25 % of capacity during a rollout is not acceptable.

  1. Pace and tolerance: minReadySeconds, progressDeadlineSeconds, revisionHistoryLimit

Three fields that complete the control over a rollout.

Field Default What it does Why it matters
minReadySeconds 0 Seconds a pod must stay ready without crashing before counting as available Stops the rollout advancing over pods that start and die after 5 seconds
progressDeadlineSeconds 600 Seconds without progress after which the rollout is declared failed Turns a silent stall into a detectable Progressing: False condition
revisionHistoryLimit 10 How many old ReplicaSets (at 0 replicas) are kept It is your rollback capability; at 0, you can undo nothing

minReadySeconds deserves a separate explanation because its effect is subtle but very valuable. With the default value of 0, as soon as a new pod says "I am ready", the controller retires another old one and carries on. If that pod crashes two seconds later, the damage is done: the rollout is advancing over a broken version. With minReadySeconds: 15, each new pod has to survive 15 seconds ready before counting. If it crashes sooner, the rollout stops.

progressDeadlineSeconds is the difference between a rollout that fails silently and one that raises the alarm. Without it, a rollout stalled by a non-existent image would keep trying forever without any alerting system noticing.

The bookings-api manifest with the three fields and the production strategy:

# k8s/base/bookings-api-deployment.yaml (updated fragment)
spec:
  replicas: 4
  revisionHistoryLimit: 5
  minReadySeconds: 15
  progressDeadlineSeconds: 300
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0

With this configuration, an update of bookings-api in rutas-norte-pro takes at least 4 × 15 = 60 seconds, and if something fails it is declared failed after 5 minutes instead of 10. Slowness in exchange for safety: on a component that handles ticket payments, that is an excellent trade.

  1. What triggers a new revision and what does not

This is the most important rule in the lesson and the most forgotten:

Only changes under spec.template trigger a new revision. Nothing else does.

The logic is transparent: a revision is a ReplicaSet, and one ReplicaSet is told apart from another by the hash of its pod template. If the template does not change, the hash does not change, and there is no new ReplicaSet to create.

Change Creates a revision? What happens
spec.template.spec.containers[].image Yes New ReplicaSet; gradual replacement of pods
spec.template.spec.containers[].env Yes Same: the pods are recreated with the new variables
spec.template.spec.containers[].resources Yes Same
spec.template.metadata.labels or annotations Yes Same, even if the container is identical
spec.template.spec.terminationGracePeriodSeconds Yes Same
spec.replicas No The current ReplicaSet raises or lowers its number of pods
spec.strategy No It will apply on the next update
spec.minReadySeconds No Same
spec.revisionHistoryLimit No It only cleans up old ReplicaSets
The Deployment's metadata.annotations No Including change-cause: it is Deployment metadata, not pod metadata

Check it for yourself:

kubectl rollout history deployment/bookings-api | tail -3
kubectl scale deployment bookings-api --replicas=5
kubectl rollout history deployment/bookings-api | tail -3
REVISION  CHANGE-CAUSE
1         Initial rollout of bookings-api 2.4.0
2         <none>

deployment.apps/bookings-api scaled

REVISION  CHANGE-CAUSE
1         Initial rollout of bookings-api 2.4.0
2         <none>

Same history: scaling is not deploying.

A practical consequence that surprises many people: changing a ConfigMap does not restart the pods that consume it, because the ConfigMap is not in the template, it is only referenced from it. The usual trick, which you will see in module 3, is to force a change in the template with a computed annotation, or to use directly:

kubectl rollout restart deployment/bookings-api

rollout restart changes nothing functional: it writes a kubectl.kubernetes.io/restartedAt annotation inside the template, which alters the hash and triggers a gradual replacement of every pod. It is the correct, downtime-free way to "restart" a Deployment.

  1. The full kubectl rollout toolkit

kubectl set image: update the image

kubectl set image deployment/bookings-api api=node:20.16-alpine
deployment.apps/bookings-api image updated

The syntax is <container-name>=<image>. The container name is the one in spec.template.spec.containers[].name, not the pod's. With several containers you can update them all at once by separating them with spaces.

A word on discipline: in a declarative project, the correct route is to edit the manifest and apply it. set image is for emergencies, and its change is lost on the next apply.

kubectl rollout status: wait

kubectl rollout status deployment/bookings-api
Waiting for deployment "bookings-api" rollout to finish: 1 out of 4 new replicas have been updated...
Waiting for deployment "bookings-api" rollout to finish: 2 out of 4 new replicas have been updated...
Waiting for deployment "bookings-api" rollout to finish: 3 out of 4 new replicas have been updated...
Waiting for deployment "bookings-api" rollout to finish: 1 old replicas are pending termination...
deployment "bookings-api" successfully rolled out

kubectl rollout history: the history

kubectl rollout history deployment/bookings-api
deployment.apps/bookings-api
REVISION  CHANGE-CAUSE
1         Initial rollout of bookings-api 2.4.0
2         Update to node 20.15 for a security patch
3         Update to node 20.16

And the full detail of one particular revision:

kubectl rollout history deployment/bookings-api --revision=2
deployment.apps/bookings-api with revision #2
Pod Template:
  Labels:  app=bookings-api
           app.kubernetes.io/part-of=rutas-norte
           environment=dev
           pod-template-hash=9f2a7c531
  Annotations:  kubernetes.io/change-cause: Update to node 20.15 for a security patch
  Containers:
   api:
    Image:  node:20.15-alpine
    Port:   3000/TCP
    Limits:  cpu: 500m, memory: 256Mi
    Requests: cpu: 100m, memory: 128Mi

That information comes entirely from the ReplicaSet at 0 replicas that was kept. That is why revisionHistoryLimit: 0 would leave you with no history and no rollback.

kubectl rollout undo: undo

kubectl rollout undo deployment/bookings-api
deployment.apps/bookings-api rolled back

It goes back to the immediately previous revision. To go to a specific one:

kubectl rollout undo deployment/bookings-api --to-revision=1

Two important clarifications:

  1. A rollback is just another deployment: it uses the same strategy, respects maxSurge and maxUnavailable, and therefore does not interrupt the service either.
  2. Undoing does not delete the bad revision: it creates a new revision with the old one's content. If you were on 3 and you undo, you end up on 4, whose content is that of 2. The numbers never go backwards.
kubectl rollout history deployment/bookings-api
REVISION  CHANGE-CAUSE
1         Initial rollout of bookings-api 2.4.0
3         Update to node 20.16
4         Update to node 20.15 for a security patch

Notice that revision 2 has disappeared from the list: its ReplicaSet has been reused as revision 4.

kubectl rollout pause and resume: freeze midway

kubectl rollout pause deployment/bookings-api

With the Deployment paused, the changes you make are not applied. It is the tool for grouping several modifications into a single rollout:

kubectl rollout pause deployment/bookings-api
kubectl set image deployment/bookings-api api=node:20.17-alpine
kubectl set resources deployment/bookings-api -c=api --limits=cpu=600m,memory=384Mi
kubectl scale deployment bookings-api --replicas=5
# so far NOTHING has happened in the cluster
kubectl rollout resume deployment/bookings-api
kubectl rollout status deployment/bookings-api
deployment.apps/bookings-api paused
deployment.apps/bookings-api image updated
deployment.apps/bookings-api resource requirements updated
deployment.apps/bookings-api scaled
deployment.apps/bookings-api resumed
Waiting for deployment "bookings-api" rollout to finish: 3 of 5 updated replicas are available...
deployment "bookings-api" successfully rolled out

Without pause, those three commands would have caused three chained rollouts, each one interrupting the previous. With pause, a single one and a single revision.

The other use of pause is for emergencies: if you see a rollout producing broken pods, kubectl rollout pause freezes it on the spot, leaving whatever old pods remain serving while you decide whether to fix or undo.

Summary table of the full toolkit:

Command What for
rollout status Wait and verify; returns a non-zero code on failure
rollout history See revisions; with --revision=N, the detail
rollout undo Go back, with or without --to-revision
rollout pause / resume Freeze and resume; group changes
rollout restart Recreate every pod without changing anything

  1. The kubernetes.io/change-cause annotation

You will have noticed the <none> entries in the CHANGE-CAUSE column. A revision history without reasons is nearly useless: three weeks later nobody remembers what revision 7 was.

kubernetes.io/change-cause is an annotation on the Deployment whose value is copied to that revision's ReplicaSet and shown in the history. Three ways to fill it in, from worst to best:

With kubectl annotate (quick, imperative):

kubectl annotate deployment/bookings-api \
  kubernetes.io/change-cause="Update to node 20.17 and raised limits for the August peaks" \
  --overwrite

In the manifest (the correct one according to the project conventions):

metadata:
  name: bookings-api
  annotations:
    kubernetes.io/change-cause: "v2.5.0 - availability cache per departure (RN-482)"

Automated in CI/CD, which is ideal:

kubectl annotate deployment/bookings-api \
  kubernetes.io/change-cause="${CI_COMMIT_SHA} · ${CI_COMMIT_MESSAGE} · ${CI_USER}" \
  --overwrite

A detail you already know from section 5, but worth repeating because it causes confusion: changing only this annotation does not trigger a new revision, because it lives in the Deployment's metadata and not in the template. Annotate it together with the real change, not afterwards.

Good practice for the text at Rutas Norte: image version, a plain-language summary of the change and the ticket reference. A real example from the project:

REVISION  CHANGE-CAUSE
1         v2.4.0 - initial rollout
2         v2.4.1 - fix to the free-seat calculation (RN-455)
3         v2.5.0 - availability cache per departure (RN-482)
4         v2.4.1 - ROLLBACK: 2.5.0 returned 500 when booking (RN-489)

  1. Zero-downtime deployment: what it really takes

Here comes the most important warning of the lesson. You might think that with a well-configured RollingUpdate you already have zero-downtime deployments. You do not, and this is the number one cause of 502 errors during supposedly perfect rollouts.

The problem is that, with what we know so far, Kubernetes considers a pod "ready" as soon as the process starts. But a Node.js process takes a few seconds to load dependencies, connect to PostgreSQL and become operational. During that window:

  1. The pod already counts as Ready.
  2. The Deployment confidently retires an old pod.
  3. The Service starts sending it real traffic.
  4. The new pod still cannot respond: 502 errors.
flowchart TD
    A["A new pod starts"] --> B["Kubernetes marks it Ready<br/>merely because the process is alive"]
    B --> C["The Service sends it traffic"]
    C --> D{"Is the application<br/>really operational?"}
    D -->|Yes| E["All correct"]
    D -->|No: still connecting to PostgreSQL| F["502s to real customers<br/>for several seconds"]

The missing piece is called a readiness probe (readinessProbe): a check the kubelet runs against your application that decides whether the pod is genuinely ready to receive traffic. With it, step 1 of the diagram does not happen until the application truly responds.

The complete checklist for a genuinely downtime-free rollout at Rutas Norte:

Requirement Status right now Where it is solved
A well-parameterised RollingUpdate strategy Done in this lesson 02-04
More than one replica Done 02-03
minReadySeconds as a cushion Done 02-04
A readinessProbe reflecting real availability Pending 07-01
A livenessProbe to detect hung pods Pending 07-01
Graceful shutdown that catches SIGTERM Done 02-01
A Service routing to the ready pods Pending 02-05
Compatibility between consecutive API versions Development's responsibility

In other words: RollingUpdate is necessary but not sufficient. Without probes, the rolling update advances blind. Until we get to module 7, minReadySeconds is a poor but useful substitute.

  1. Diagnosing and rolling back a stalled deployment

The final exercise and the most realistic of all. One Friday afternoon, somebody deploys a version to rutas-norte-dev that does not exist in the registry.

The disaster

kubectl annotate deployment/bookings-api \
  kubernetes.io/change-cause="v2.6.0 - version that does not exist (RN-501)" --overwrite
kubectl set image deployment/bookings-api api=node:99.99-nonexistent
kubectl rollout status deployment/bookings-api --timeout=90s
deployment.apps/bookings-api annotated
deployment.apps/bookings-api image updated
Waiting for deployment "bookings-api" rollout to finish: 1 out of 5 new replicas have been updated...
error: timed out waiting for the condition

Step 1: the overall state

kubectl get deploy bookings-api
NAME           READY   UP-TO-DATE   AVAILABLE   AGE
bookings-api   5/5     1            5           1h

Reading: there are 5 ready and available pods —the service is still running on the previous version— but only 1 has the new version. The rollout took one step and stopped. This is the system working properly: maxUnavailable: 0 prevented any old pod being retired until the new one was ready, and since it never will be, none has been retired.

Step 2: the ReplicaSets

kubectl get rs -l app=bookings-api
NAME                     DESIRED   CURRENT   READY   AGE
bookings-api-3d8e1f602   1         1         0       2m
bookings-api-c5a9b47d8   5         5         5       18m

The new ReplicaSet has 1 pod that never reaches READY.

Step 3: the guilty pod

kubectl get pods -l app=bookings-api | grep -v Running
NAME                           READY   STATUS             RESTARTS   AGE
bookings-api-3d8e1f602-h4tqm   0/1     ImagePullBackOff   0          2m
kubectl describe pod bookings-api-3d8e1f602-h4tqm | tail -6
Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Pulling    2m                  kubelet            Pulling image "node:99.99-nonexistent"
  Warning  Failed     2m                  kubelet            Failed to pull image: manifest for node:99.99-nonexistent not found
  Warning  Failed     2m                  kubelet            Error: ErrImagePull
  Normal   BackOff    30s (x6 over 2m)    kubelet            Back-off pulling image "node:99.99-nonexistent"

Root cause identified, using exactly the procedure from the Pods lesson.

Step 4: the conditions

kubectl describe deployment bookings-api | grep -A4 Conditions
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    False   ProgressDeadlineExceeded

Available: True (we are still serving) + Progressing: False with ProgressDeadlineExceeded (the rollout is dead) is the exact signature of a stalled deployment. In a monitored cluster, that combination should raise an alert.

Step 5: decide and act

Two options, depending on the moment:

# Option A: freeze while investigating (does not revert, only stops)
kubectl rollout pause deployment/bookings-api

# Option B: undo right now (the normal move on a Friday afternoon)
kubectl rollout undo deployment/bookings-api
kubectl rollout status deployment/bookings-api
deployment.apps/bookings-api rolled back
deployment "bookings-api" successfully rolled out

An important warning: if the Deployment is paused, undo has no effect. You have to resume first.

Step 6: verify and document

kubectl get deploy,rs -l app=bookings-api
kubectl annotate deployment/bookings-api \
  kubernetes.io/change-cause="ROLLBACK to v2.5.0: the v2.6.0 image does not exist in the registry (RN-501)" \
  --overwrite
kubectl rollout history deployment/bookings-api
NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/bookings-api   5/5     5            5           1h

NAME                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/bookings-api-3d8e1f602   0         0         0       6m
replicaset.apps/bookings-api-c5a9b47d8   5         5         5       22m

REVISION  CHANGE-CAUSE
1         Initial rollout of bookings-api 2.4.0
4         v2.6.0 - version that does not exist (RN-501)
5         ROLLBACK to v2.5.0: the v2.6.0 image does not exist in the registry (RN-501)

The broken ReplicaSet stays at 0 replicas, consuming nothing, as a record of the incident. And the history tells the whole story to whoever reads it in six months' time.

The moral of the whole section: during those six steps, bookings-api did not miss a single booking. With the original Docker Compose setup, that same mistake would have left the platform down until somebody noticed. That is what a well-configured Deployment buys you.

Leave the cluster tidy before moving on:

kubectl scale deployment bookings-api --replicas=2
kubectl get deploy
NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
bookings-api           2/2     2            2           1h
redis-cache            1/1     1            1           50m
web-store              3/3     3            3           1h
notifications-worker   2/2     2            2           45m

Common Mistakes and Tips

  • Believing that RollingUpdate guarantees zero downtime on its own. Without a readinessProbe, traffic reaches pods that cannot respond yet. It is the number one cause of 502s during rollouts.
  • Setting maxSurge: 0 and maxUnavailable: 0. The API rejects it: the rollout could not advance.
  • Using RollingUpdate with a workload that cannot tolerate two simultaneous versions. Single-replica databases, processes with an exclusive lock or incompatible schema migrations: those call for Recreate, or a StatefulSet outright.
  • Setting revisionHistoryLimit: 0 to "save resources". ReplicaSets at 0 replicas consume neither CPU nor memory, just a few kilobytes in etcd. In exchange, you lose the ability to undo entirely.
  • Expecting a change to spec.strategy or spec.replicas to trigger a rollout. They are not in the template. They only apply on the next real update.
  • Expecting a change to a ConfigMap to restart the pods. It does not. Use kubectl rollout restart.
  • Forgetting change-cause. A history full of <none> is no use at all when you have to decide which revision to go back to at three in the morning.
  • Annotating the change-cause after the rollout. Since it does not trigger a revision, it stays attached to the Deployment but may not reflect what that revision did. Annotate it in the same apply.
  • Running undo on a paused Deployment. Nothing happens until you resume. It is a classic of blind debugging.
  • Trusting kubectl apply as verification. apply only says the API accepted the object. Always chain kubectl rollout status --timeout=... in your scripts.
  • Tip: in production, maxSurge: 1 and maxUnavailable: 0. It costs one extra pod of capacity and guarantees you never drop below nominal capacity.
  • Tip: kubectl rollout status returns a non-zero exit code if the rollout fails. That is what makes a CI/CD pipeline trustworthy: kubectl apply -f k8s/ && kubectl rollout status deploy/bookings-api --timeout=300s || kubectl rollout undo deploy/bookings-api.

Exercises

Exercise 1: Work out a rollout on paper

web-store is in rutas-norte-pro with 6 replicas, maxSurge: 2 and maxUnavailable: 1.

  1. What is the maximum number of simultaneous pods and the minimum number of available pods?
  2. Build a table with the first five steps of the update, showing for each one: old pods, new pods, total and available.
  3. If the platform team demands that the number of serving pods never drops below 6, what values would you set and what extra resources does the cluster need?
  4. With the configuration from point 3 and minReadySeconds: 20, what is the theoretical minimum duration of the rollout if each pod takes 5 seconds to start?

Exercise 2: A complete update and rollback cycle

On web-store in rutas-norte-dev:

  1. Configure RollingUpdate with maxSurge: 1, maxUnavailable: 0, minReadySeconds: 10 and revisionHistoryLimit: 5, applying it from the manifest.
  2. Update to nginx:1.27.1-alpine with its corresponding change-cause and wait for it to finish.
  3. Update to nginx:1.26-alpine with another change-cause.
  4. Show the history and the detail of revision 2.
  5. Pretend 1.26 has a serious bug: go back to 1.27.1 and verify that all three replicas serve it.
  6. Explain why the history does not show the numbering you would expect.

Exercise 3: Diagnose a stalled rollout of notifications-worker

  1. Break notifications-worker on purpose by deploying the image busybox:9.9.9-nonexistent, with a change-cause included.
  2. With four commands, diagnose the problem following the correct order: Deployment state, ReplicaSets, guilty pod and conditions.
  3. Answer: are the confirmation emails still being sent during the stall? Justify it with the commands' output.
  4. Freeze the rollout, check that undo does nothing while it is paused, resume it and undo.
  5. Document the rollback with an annotation and show the final history.

Solutions

Solution 1

  1. Maximum pods = replicas + maxSurge = 6 + 2 = 8. Minimum available = replicas - maxUnavailable = 6 − 1 = 5.

  2. The first five steps:

Step Action Old New Total Available
0 Initial state 6 ready 0 6 6
1 Creates 2 new pods (maxSurge ceiling) 6 ready 2 being created 8 6
2 Retires 1 old one (maxUnavailable ceiling) 5 ready 2 being created 7 5
3 The 2 new ones become Ready 5 ready 2 ready 7 7
4 Creates 1 more new one (7 + 1 = 8, the ceiling) 5 ready 2 ready + 1 being created 8 7
5 Retires 2 old ones (available 7 − 2 = 5, the floor) 3 ready 2 ready + 1 being created 6 5
  1. To never drop below 6 serving: maxUnavailable: 0, and maxSurge at 1 or 2 depending on how fast you want to go. With maxSurge: 2, the cluster must have room for 8 web-store pods at once: with requests of 50m of CPU and 64Mi per pod, that is an extra 100m of CPU and 128Mi of memory free during the rollout. With maxUnavailable: 0 the cluster must have that capacity; if it does not, the new pods stay Pending and the rollout never advances.

  2. With maxSurge: 2 and maxUnavailable: 0, pods are replaced in batches of 2. Each batch takes 5 s (startup) + 20 s (minReadySeconds) = 25 s. Six pods in batches of two make 3 batches: 75 seconds as a theoretical minimum, not counting image pulls or the old pods' termination time.

Solution 2

# k8s/base/web-store-deployment.yaml (fragment)
spec:
  replicas: 3
  revisionHistoryLimit: 5
  minReadySeconds: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
kubectl apply -f k8s/base/web-store-deployment.yaml

# 2. Update to 1.27.1
kubectl annotate deployment/web-store \
  kubernetes.io/change-cause="v1.27.1 - nginx security patch (RN-470)" --overwrite
kubectl set image deployment/web-store nginx=nginx:1.27.1-alpine
kubectl rollout status deployment/web-store

# 3. Update to 1.26
kubectl annotate deployment/web-store \
  kubernetes.io/change-cause="v1.26 - back to the previous branch for compatibility (RN-473)" --overwrite
kubectl set image deployment/web-store nginx=nginx:1.26-alpine
kubectl rollout status deployment/web-store
deployment "web-store" successfully rolled out
# 4. History and detail
kubectl rollout history deployment/web-store
kubectl rollout history deployment/web-store --revision=2
deployment.apps/web-store
REVISION  CHANGE-CAUSE
1         Initial rollout of web-store 1.27.0
2         v1.27.1 - nginx security patch (RN-470)
3         v1.26 - back to the previous branch for compatibility (RN-473)

deployment.apps/web-store with revision #2
Pod Template:
  Labels:  app=web-store
           pod-template-hash=6f9c4b8d7
  Annotations:  kubernetes.io/change-cause: v1.27.1 - nginx security patch (RN-470)
  Containers:
   nginx:
    Image:  nginx:1.27.1-alpine
# 5. Rollback to revision 2
kubectl rollout undo deployment/web-store --to-revision=2
kubectl rollout status deployment/web-store
kubectl get pods -l app=web-store \
  -o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image
deployment.apps/web-store rolled back
deployment "web-store" successfully rolled out

NAME                         IMAGE
web-store-6f9c4b8d7-42kxr    nginx:1.27.1-alpine
web-store-6f9c4b8d7-8vnwq    nginx:1.27.1-alpine
web-store-6f9c4b8d7-t9mzd    nginx:1.27.1-alpine
# 6. Final history
kubectl rollout history deployment/web-store
REVISION  CHANGE-CAUSE
1         Initial rollout of web-store 1.27.0
3         v1.26 - back to the previous branch for compatibility (RN-473)
4         v1.27.1 - nginx security patch (RN-470)
  1. The numbering is surprising because revisions never go backwards. When you undo to revision 2, Kubernetes does not "return" to it: it reuses its ReplicaSet and renumbers it as revision 4, the most recent. That is why 2 disappears from the list and a 4 appears with the same change-cause. The revision number identifies the order in which the changes were applied, not the content: a rollback is just another change.

Solution 3

# 1. Break it
kubectl annotate deployment/notifications-worker \
  kubernetes.io/change-cause="v1.3.0 - image that does not exist (RN-512)" --overwrite
kubectl set image deployment/notifications-worker worker=busybox:9.9.9-nonexistent
# 2. Diagnosis in four steps
kubectl get deploy notifications-worker
kubectl get rs -l app=notifications-worker
kubectl get pods -l app=notifications-worker
kubectl describe pod <pod-in-ImagePullBackOff> | tail -6
kubectl describe deployment notifications-worker | grep -A4 Conditions
NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
notifications-worker   2/2     1            2           1h

NAME                              DESIRED   CURRENT   READY   AGE
notifications-worker-5a3f9d21c    1         1         0       90s
notifications-worker-8c7b5d94f    2         2         2       1h

NAME                                   READY   STATUS             RESTARTS   AGE
notifications-worker-5a3f9d21c-r6bkt   0/1     ImagePullBackOff   0          90s
notifications-worker-8c7b5d94f-k2xrt   1/1     Running            0          1h
notifications-worker-8c7b5d94f-p9mzq   1/1     Running            0          1h

  Warning  Failed   80s   kubelet   Failed to pull image "busybox:9.9.9-nonexistent": manifest unknown

Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    False   ProgressDeadlineExceeded
  1. Yes, the emails are still going out. The proof is in three places: READY 2/2 and AVAILABLE 2 on the Deployment, the 2 pods of the previous revision in Running, and above all Available: True with MinimumReplicasAvailable in the conditions. Only UP-TO-DATE 1 gives away that a new version is trying to get in and failing. Direct confirmation:
kubectl logs -l app=notifications-worker --tail=1 --prefix | grep sending
[pod/notifications-worker-8c7b5d94f-k2xrt/worker] sending batch of confirmations from notifications-worker-8c7b5d94f-k2xrt
# 4. Pause, check that undo does nothing, resume and undo
kubectl rollout pause deployment/notifications-worker
kubectl rollout undo deployment/notifications-worker
kubectl get rs -l app=notifications-worker
deployment.apps/notifications-worker paused
deployment.apps/notifications-worker rolled back

NAME                              DESIRED   CURRENT   READY   AGE
notifications-worker-5a3f9d21c    1         1         0       4m
notifications-worker-8c7b5d94f    2         2         2       1h

The broken ReplicaSet still has 1 replica: the undo has been recorded but not executed, because the Deployment is paused.

kubectl rollout resume deployment/notifications-worker
kubectl rollout status deployment/notifications-worker
kubectl get rs -l app=notifications-worker
deployment.apps/notifications-worker resumed
deployment "notifications-worker" successfully rolled out

NAME                              DESIRED   CURRENT   READY   AGE
notifications-worker-5a3f9d21c    0         0         0       5m
notifications-worker-8c7b5d94f    2         2         2       1h
# 5. Document
kubectl annotate deployment/notifications-worker \
  kubernetes.io/change-cause="ROLLBACK to v1.2.0: the v1.3.0 image does not exist in the registry (RN-512)" \
  --overwrite
kubectl rollout history deployment/notifications-worker
REVISION  CHANGE-CAUSE
1         Initial rollout of notifications-worker 1.2.0
2         v1.3.0 - image that does not exist (RN-512)
3         ROLLBACK to v1.2.0: the v1.3.0 image does not exist in the registry (RN-512)

Conclusion

You have closed the circle we opened in the previous lesson with that second ReplicaSet that appeared without explanation. You now know that the Deployment's two native strategies are RollingUpdate and Recreate, and why the choice is not a matter of taste but of the workload's nature: web-store, bookings-api and notifications-worker tolerate two versions coexisting and go with RollingUpdate; redis-cache does not, and goes with Recreate; and bookings-postgres takes neither, because a Deployment is not where it belongs: two PostgreSQL processes over the same data directory mean a lock-out in the best case and corruption in the worst.

You can work out a rollout on paper before running it: replicas + maxSurge is the ceiling of pods and replicas - maxUnavailable the floor of available ones, and with those two numbers you can reconstruct step by step what the controller will do with the 4 replicas of bookings-api. You know the useful combinations and the one recommended for production, maxSurge: 1 with maxUnavailable: 0, which protects nominal capacity on an August peak. You tune the pace with minReadySeconds, you detect stalls with progressDeadlineSeconds and you preserve your ability to undo with revisionHistoryLimit. You are clear on the rule that governs everything: only changes under spec.template create a revision, which is why scaling does not deploy, changing a ConfigMap restarts nothing and kubectl rollout restart exists. And you handle the full toolkit —status, history, undo, pause, resume— with change-cause documenting every step, including the rollback of a deployment stalled by a non-existent image, resolved without a single customer failing to buy their ticket.

One warning and one big gap remain. The warning: RollingUpdate is necessary but not sufficient for genuinely downtime-free deployments; without a readinessProbe, Kubernetes marks a pod whose process has just started as ready and sends it traffic before it can serve it. That piece arrives in Health Checks and Probes. And the gap: all the traffic we have spent two lessons talking about does not exist yet. Our pods have IPs that change on every rollout, and to talk to bookings-api we have had to look up a specific pod's IP by hand. In the next lesson, Services, we will give each Rutas Norte component a stable address that survives rollouts, load balancing across replicas included.

Kubernetes Course

Module 1: Introduction to Kubernetes

Module 2: Core Kubernetes Components

Module 3: Configuration and Secret Management

Module 4: Networking in Kubernetes

Module 5: Storage in Kubernetes

Module 6: Advanced Kubernetes Concepts

Module 7: Monitoring and Logging

Module 8: Kubernetes Security

Module 9: Scaling and Performance

Module 10: Kubernetes Ecosystem and Tooling

Module 11: Case Studies and Real-World Applications

Module 12: Preparing for Kubernetes Certification

© Copyright 2026. All rights reserved