Aurora Libros scales by itself and takes the peak. What is left is the most delicate moment of all: releasing a new version. Between the old one and the new one there is always a period in which they coexist, and deciding how that period is managed is the difference between a release nobody notices and fifteen minutes during which nobody can buy a book.

1. The problem: the coexistence window
2. The five strategies, compared
3. Rolling updates in Kubernetes: maxSurge and maxUnavailable
4. update_config in Swarm and the mapping table
5. Readiness and minReadySeconds
6. Watching a deployment in progress
7. Rollback: undo, revisions and revisionHistoryLimit
8. Rollback in Swarm
9. Rehearsing a failed deployment: aurora-api:2.1.0
10. Blue-green with two Deployments
11. Canary by replicas and by Ingress
12. What you watch during a canary
13. Argo Rollouts, Flagger and GitOps
14. Database migrations: expand/contract
15. Feature flags

During a rolling deployment, traffic is split between different versions. That forces version 2.1.0 to be compatible with 2.0.0 on three fronts:

The third is the most surprising and the easiest to trigger: if 2.1.0 changes the format of what it stores in aurora-cache under the same key, the old replicas will read objects they do not understand and return intermittent errors that come and go depending on which Pod each request lands on. The fix is as simple as versioning the key: books:v2:*.

Shadow deserves a note: it duplicates real traffic towards the new version but discards its response, so the user never sees it. It is magnificent for validating performance with authentic load, and it has a dangerous trap: if the duplicated request writes to the database or charges a card, the operation happens twice. It only works with read traffic or with isolated dependencies.

The combination maxSurge: 2 / maxUnavailable: 0 is the right one for Aurora Libros: there are never fewer than six Pods serving, so capacity does not drop for a single instant during the release. It costs two extra Pods for a few minutes, which is a ridiculous price. maxUnavailable: 0 is also an insurance policy: if the new Pods never become ready, the deployment cannot advance, because advancing would require removing old Pods.

Watch out for one specific case: maxSurge: 0 and maxUnavailable: 0 together is an invalid configuration —the deployment could not take a single step— and Kubernetes rejects it.

Two important differences. Swarm can undo by itself with failure_action: rollback, whereas Kubernetes limits itself to stopping and waiting for you to decide —which is defensible: stopping leaves the system in a known state and does not hide the problem. In exchange, Swarm keeps only the immediately previous version, while Kubernetes preserves as many revisions as you tell it to.

This is where what you did in 06-01 really pays off. Without a readiness probe, a rolling update is a game of roulette: Kubernetes considers a Pod "ready" the moment its container starts, sends it traffic, and your application is still establishing the PostgreSQL pool. Every replaced replica produces a few seconds of errors.

minReadySeconds: 10 adds a second safeguard: it requires the Pod to have been ready for ten consecutive seconds before counting it as available and moving on to the next. Without it, a Pod that passes readiness and falls over two seconds later would still advance the deployment, and you would end up with six broken replicas. With it, the deployment stops at the first unstable Pod.

And the preStop with sleep 5 from 06-05 closes the other end: Pods on their way out stop receiving traffic before they close their connections. Readiness on the way in, preStop on the way out: between the two, zero errors during a release.

The two ReplicaSets are the key to everything that follows. The old one stays at zero replicas, it is not deleted: it is the history that makes undo possible. The rollout status returns code 0 if it finishes and non-zero if it fails, which makes it the natural gate for the last step of the pipeline from 06-02.

Look at revision 5: the rollback does not delete number 4, it creates a new revision with the contents of number 3. History always moves forward, so you can move ahead again once you fix the problem.

revisionHistoryLimit: 5 in your manifest controls how many old ReplicaSets are kept. A value of 0 makes any undo impossible; an enormous value fills the namespace with empty objects. Somewhere between 5 and 10 is sensible.

And the golden rule, which is not a command: the rollback has to be tested before you need it. An undo nobody has ever run is an assumption, not a plan. Rehearse it in staging every time you change something relevant about the deployment, because the day you need it will be three in the morning with customers waiting.

With failure_action: rollback, Swarm does it on its own: if a new task fails during the monitor window and max_failure_ratio is exceeded, it reverts with nobody intervening. It is convenient and it has a clear limit: it only keeps the previous specification, so a rollback on top of another rollback puts you back where you started.

Let's simulate the real case: 2.1.0 has a typo in the name of the cache variable, so readiness never passes.

There is the whole mechanism: six old Pods serving and two new ones that never reach 1/1. The deployment stopped by itself at the second Pod because maxUnavailable: 0 forbids removing any of the old ones while the new ones are not ready. No client has noticed anything; the two broken Pods receive no traffic because their readiness returns 503 and kube-proxy does not have them in the endpoints.

This is the scenario that justifies every decision in the module: the readiness probe detected the problem, maxUnavailable: 0 stopped it from spreading and the revision history let you undo it in one command.

The idea is to have two complete Deployments and a Service whose selector decides which one receives the traffic.

This is what the label-based coupling from 06-04 enabled. The rollback is instant —a selector change, with nothing to start— and that is its great virtue. The cost is equally obvious: during the transition you pay twice the resources, and with twelve API replicas that is not trivial. On top of that, the connections already open against the blue Pods stay there until they finish; the switch is not as atomic as it looks.

A practical rule: keep the blue environment alive for at least one full business cycle —a peak hour, a day— before removing it. A cheap rollback only exists while the other environment is still standing.

By replicas, with no tooling: two Deployments with the same selector label and the Service distributing by Pod count.

It is approximate —kube-proxy's distribution is random— and coarse-grained: serving 1 % would take 99 stable replicas. But it requires installing nothing.

By Ingress, with real precision:

The canary-by-header annotation is more useful than it looks: it lets the team push its own traffic to the canary with a header, and validate the new version with known real users before opening the percentage to the public.

A canary with no metrics is just a slow deployment. What turns it into a strategy is comparing the two versions using the same indicators over the same window.

The waiting criterion matters as much as the thresholds. At three in the morning, 10 % of the traffic for five minutes is thirty requests: they tell you absolutely nothing. That is why the condition has to be double, on time and on volume. And the promotion, as soon as it is feasible, should be automatic against those thresholds: if it depends on somebody watching a dashboard, it will end up being promoted without anybody looking.

The GitOps model reverses the direction of the deployment and is worth understanding: instead of the pipeline running kubectl apply against the cluster (push), an agent inside the cluster watches the repository and applies what it finds (pull). The advantages are concrete: CI does not need cluster credentials —the risk of that last step in 06-02 disappears—, the desired state lives in Git with its history and its revisions, and any manual change is detected as drift and corrected. The rollback becomes a git revert.

No deployment strategy solves this on its own, because the schema is shared by every version at once.

Imagine 2.1.0 adds language to the books table. The naive approach —deploying code and schema together— fails in both directions: if you migrate first, the old replicas do not know about the column; if you deploy first, the new ones fail because the column does not exist. And if you roll back the code, what do you do with the schema?

The answer is the expand/contract pattern, in three separate deployments:

2.0.0 keeps working: it ignores a column it does not know about. Only the schema is deployed, without touching the code.

Now 2.1.0 does get deployed, already using language. During the rolling update both versions coexist without any problem: the old one does not touch the column, the new one fills it in. And the rollback is safe, because going back to 2.0.0 breaks nothing.

The rule that follows: schema and code are never deployed coupled together. The schema always goes first and additively; the removal of the old thing always goes last and one cycle behind. That way, at every moment, the database is compatible with the previous version and with the next one, and any code rollback is safe.

Warning. A migration over real data can lock tables, exhaust the disk through the WAL or make a deployment irreversible. An ALTER TABLE that rewrites a large table leaves the service blocked for as long as it runs. Always validate the migration plan, the window and the fallback procedure with the data officer in your organization, and rehearse it first against a copy of the production volume.

The last piece separates two things we tend to confuse: deploying code and enabling a feature.

With the flag off, the 2.1.0 code gets deployed all the way to production and does nothing. It is enabled afterwards, whenever you want, by changing a ConfigMap and without deploying anything; and if it goes badly, it is switched off in seconds, which is infinitely faster than any image rollback.

The price is technical debt: every flag is a live branch in the code and two paths to test. They are set with an expiry date and removed as soon as the feature is settled, or you end up with twenty flags and not one tested combination.

• Deploying with no readiness probe. Every replaced replica produces a few seconds of errors. It is the prerequisite for everything else.
• maxUnavailable > 0 with tight capacity. You release precisely when you have the least capacity. Use maxSurge and maxUnavailable: 0.
• Schema and code in the same deployment. It blocks the rollback: you can go back on the code, but not on the schema. Expand/contract, always.
• A canary with no metrics and no criteria. It is a slow deployment with more steps. Define thresholds and a window before you start.
• revisionHistoryLimit: 0. It leaves kubectl rollout undo with nothing to go back to.
• Blue-green and deleting blue right away. You lose the only thing that made the rollback cheap. Wait a full cycle.
• Changing the cache format without versioning the key. Intermittent errors that are impossible to reproduce during coexistence.
• Tip: rehearse the rollback in staging regularly. An untested procedure is not a procedure.
• Tip: make the pipeline fail if kubectl rollout status fails. A green deployment with broken Pods is the worst of both worlds.

Exercise 1. Rehearse a failed deployment of aurora-api:2.1.0 that never passes readiness: watch it stop by itself, check that no client receives errors throughout the whole process, and undo it while verifying the revision history.

Exercise 2. Implement a complete blue-green: deploy the green version, validate it without exposing it, switch the Service over, measure the switching time and go back.

Exercise 3. Apply expand/contract to add the language column to books with no downtime, demonstrating that a code rollback remains safe at every phase.

Solution 1.

Three numbers demonstrate everything. There are eight Pods running but only six endpoints: the two 2.1.0 Pods exist and never entered the balancing because their readiness never returned 200. And the watcher recorded 1,983 responses, every one of them a 200, during a deployment that failed completely.

The mechanism has three linked pieces, and all three were decided in earlier lessons. The readiness probe from 06-01 detected that 2.1.0 could not serve; kube-proxy excluded it from the endpoints, so it received not a single request; and maxUnavailable: 0 prevented any old Pod from being removed, because doing so would have dropped below six available. The deployment froze in a safe state.

Zero errors in the whole exercise: failed deployment, waiting and rollback included. Notice that the undo was practically instant, and the reason is that it started nothing: the six 2.0.0 Pods never stopped existing, so undoing consisted of deleting the two broken Pods and returning the old ReplicaSet to six desired replicas. A rollback after a successful deployment does take time, because the old Pods have to be recreated.

And the code: 1 from rollout status is the piece the pipeline in 06-02 was missing: all it takes is not ignoring that exit code for a deployment like this to turn the job red and trigger the undo automatically.

Solution 2.

An important detail in the first number: even though the green Deployment is already ready, the Service still has six endpoints, blue's. The new version is alive, validated through port-forward and serving the nine titles, and it receives not a single user request. That separation between "deployed" and "exposed" is the essence of blue-green.

The 412 milliseconds are the entire argument for blue-green, and the 12 in the last column is its bill. The switch is that fast because it starts and stops nothing: it only rewrites a selector, and the endpoints controller recalculates which Pods belong to the Service.

Two nuances the table does not show. First: the HTTP keep-alive connections already established against blue Pods keep being served by the old version until they close, so the switch is not atomic from the user's point of view. Second, more serious: while the two environments coexist, both versions are writing to the same database and the same cache, which means the compatibility requirement from section 1 still applies. Blue-green solves the routing, not the data compatibility.

Solution 3.

2.0.0 keeps returning the nine titles with a new column it completely ignores, because its SELECT names the columns it needs. That is the property that makes phase 1 safe: an additive change with a default value is invisible to the old code.

Those two lines are the demonstration that was asked for. On 2.1.0, /books includes the language; after the rollback to 2.0.0, the field disappears from the response and everything else keeps working. The database never had to change at any point, because the column was compatible with both versions.

The decisive column is the third: in none of the three phases do you need to undo the schema, and that is exactly the point of the pattern. Compared with the coupled approach, the difference is enormous: had you deployed the ALTER TABLE ... NOT NULL alongside 2.1.0, the code rollback would have left 2.0.0 inserting rows with no language against a mandatory column, and every insert would have failed.

Phase 3 demands a discipline that is easily forgotten: it cannot be run until you are sure no old version is still alive, and that includes batch jobs, reporting processes or third-party integrations that nobody may remember. That is why the prudent thing is to let a full cycle go by —days, not minutes— between phase 2 and phase 3. And it is why the migration plan, the window and the fallback procedure are agreed with the data officer before touching production.

You now know how to release without anybody losing the ability to buy a book. You are clear about the underlying problem —throughout every deployment two versions coexist over the same database and the same cache— and about the five strategies with their real cost: recreate with its downtime, the rolling update as the sensible default, blue-green with its 412-millisecond switch in exchange for double the resources, the canary as the lowest-risk option when you have reliable metrics, and shadow with its warning about duplicated side effects.

You have mastered the rolling update from the inside: maxSurge: 2 with maxUnavailable: 0 so capacity never drops for an instant, minReadySeconds so a Pod that falls over two seconds later is not counted as good, progressDeadlineSeconds as the limit, and the mapping table against Swarm's update_config. Above all, you have seen why the readiness probe from 06-01 is the piece that holds everything else up: the rehearsal of the failed aurora-api:2.1.0 deployment ended with eight Pods running, only six endpoints and 1,983 responses, all of them 200, with the deployment frozen in a safe state and a rollout status returning code 1 for the pipeline to catch. The undo was instant because there was nothing to start, and the revision history always moves forward, it never deletes.

You have set up blue-green by switching a selector —the label-based coupling from 06-04 paying off—, the canary by replicas and by Ingress annotations with its header for the team, and you know what to watch during a canary and by which double criterion of time and volume it gets promoted or aborted. You know about Argo Rollouts and Flagger for automating it, and about GitOps with Argo CD or Flux reversing the direction of the deployment so CI does not need cluster credentials. And you have solved what no strategy fixes on its own: the expand/contract pattern in three deployments, applied to adding language to books, with the demonstration that a code rollback is safe at every phase because the schema is never deployed coupled to it. Rounding off the picture are feature flags, which separate deploying from enabling and turn a minutes-long rollback into a seconds-long one.

And with that, module 6 closes. Aurora Libros came in as a Compose stack on your laptop and leaves as a platform: a production-grade 2.0.0 image with configuration on the outside, startup validation that fails in 0.4 seconds, a graceful shutdown that loses not a single request and three probes with separate semantics; a pipeline that on every git push tests against a real PostgreSQL and Redis, builds for two architectures in 41 seconds thanks to the remote cache, scans, signs with Cosign and publishes; a cluster —first Swarm with its overlay networks and its routing mesh, then Kubernetes with the four services in real manifests, a StatefulSet for the data, Kustomize per environment and /books returning the nine titles—; a HorizontalPodAutoscaler that took the p95 from 1,840 ms to 88 ms without anybody touching a thing, with PgBouncer solving the bottleneck that sat where you were not looking; and, now, releases with no downtime and a tested way back. Aurora Libros no longer lives on your machine: it lives in a cluster, deploys itself from a commit, grows with the load and updates without any customer noticing.

In module 7 we lift our eyes from the project to look around. You will see how the hosts that hold all of this up are provisioned, when Compose is the right call and when Kubernetes is, with an honest comparison of the two, what Docker Desktop brings and what it costs, which third-party tools and plugins deserve a place in your workflow, how Podman, containerd and the OCI standard fit in —those images of yours that you now know are not "Docker's"— and where the container ecosystem is heading.