The pipeline of 05-03 ends in kubectl rollout status and in a commit that Argo CD applies. Between "apply" and "deployed" something happens that we have taken for granted so far: the orders-service:1.0.0 replicas have to give way to the 1.0.1 ones without a single POST /v1/orders receiving a connection error. When deployments happened every two weeks in the small hours, a 30 s outage was tolerated; when each service deploys several times a day during business hours (which is the DORA target of 05-03), any outage multiplied by six services and by N deployments is unacceptable. This lesson explains the four strategies (recreate, rolling, blue-green, canary), their implementation in Kubernetes with the manifests of 05-02, the condition they all share (two versions coexist, so contracts and database schemas must be backward compatible), feature flags as a way to separate deployment from release, and the strategy TechCorp picks for each service. How the mesh does the canary by exact weight is left for 05-05.

1. Zero-downtime deployment as a requirement
2. Recreate versus RollingUpdate
3. Backward compatibility during the rolling update: contracts and expand/contract
4. Blue-green
5. Canary
6. Feature flags: deployment is not release
7. Comparison table
8. TechCorp's strategy per service
9. Deploying changes to event consumers
10. Rollback, forward-fix and progressive automation

A zero-downtime deployment demands three things we already have and one new one:

• That the new replicas receive no traffic until they are ready: readinessProbe on /health/ready (05-02).
• That the old replicas finish what they have in hand before dying: SIGTERM → /health/ready at 503 → server.close() (04-02, 05-01) within terminationGracePeriodSeconds: 30 (05-02).
• That there is more than one replica, so that one is always left serving.
• And the new one: that the old and the new version can coexist for seconds, minutes or days, talking to the same clients, the same database and the same queues (sections 3 and 9).

Without the fourth condition, no strategy works: there is no way to change version "instantly" in a distributed system; there is always a window with both.

Recreate stops all the old replicas and then starts the new ones: there is a gap of seconds with no service. It only makes sense when two versions cannot coexist (a destructive schema change, an exclusive consumer of a queue) and the outage is accepted. RollingUpdate, the default for a Deployment, replaces replicas little by little:

With replicas: 3, maxSurge: 1, maxUnavailable: 0, the sequence is:

The four parameters read like this:

The relationship with terminationGracePeriodSeconds: when the Deployment deletes a v1 pod, the kubelet sends it SIGTERM and, in parallel, the Endpoints controller removes it from the Service. There is a race of milliseconds in which a request may still arrive; that is why /health/ready switches to 503 first and server.close() keeps serving the open connections. A preStop: { exec: { command: ["sleep", "5"] } } on the container gives extra margin if errors are observed in that window.

And the rollback:

rollout undo is immediate and safe if 1.0.0 can coexist with what 1.0.1 left in the database: section 3. With GitOps (05-03), the canonical rollback is reverting the commit in platform; rollout undo is the emergency route.

During the rolling update (and for hours or days in a canary), Orders 1.0.0 and 1.0.1 serve requests at the same time, publish events at the same time and write to the same table. Two consequences:

1. Contracts: the API and the events of 1.0.1 must be backward compatible with the current consumers (03-06: adding fields yes, removing or changing type no; /v2/ for the rest). A client that talks to 1.0.0 in one request and to 1.0.1 in the next (the Service balances per connection) must not notice incompatible differences.
2. Database schema: the 1.0.1 migration is applied before its pods start (the Job of 05-02) and while 1.0.0 is still writing. So the migration has to be compatible with 1.0.0.

The pattern to achieve it is expand/contract: a schema change is made in two (or three) deployments, never in one. With the example of 03-06, adding currency to the order lines:

What is never done in a single step: renaming a column (unit_price → price), changing its type, dropping it, or adding a NOT NULL without a DEFAULT. Any of those breaks the old version the instant the Job finishes, before the new one is ready, and makes rollout undo impossible. Luis's practical rule: "a migration can only add; removing is another deployment, and it must be possible to undo the deployment without undoing the migration".

Two complete, identical environments, blue (active) and green (new). The new version is deployed to green, tested with internal traffic, and the routing is switched in one go. If something fails, you go back to blue in a second. In Kubernetes, two Deployments and a Service whose selector decides which one receives traffic:

Advantages: instant switch and rollback, and no mixed versions in the Service (although in-flight requests on blue finish on blue). Drawbacks: double cost while they coexist, and the same data-compatibility demands of section 3, because green already wrote to the database before going back to blue. When what is exposed is an Ingress, the switch can be the Ingress's backend.service.name instead of the selector.

A canary sends a small percentage of traffic to the new version, its metrics are observed (here two are enough: 5xx error rate and latency; the details of what to measure and how to alert belong to 06-01 and 06-05), and if they are equal to or better than the current version's the percentage is increased up to 100%; if not, it is withdrawn without most users having noticed. Three ways to do it, from most basic to most precise:

(a) By replicas. The Service selects by app: orders-service without looking at version (as in 05-02); a second Deployment with one replica of the new version joins the same group:

Simple and with no new moving parts; but the percentage is approximate (kube-proxy spreads per connection, not per request) and the granularity is "one replica": with 3 stable replicas, the minimum canary is 25%.

(b) By weight in the NGINX Ingress. For what comes in through the Ingress (at TechCorp, the gateway; or a service if it were exposed directly), a second Ingress with canary annotations:

Exact per-request weight and canary by header (X-Canary), which lets the team try the new version in production before opening up the percentage: the same idea as the router of the Express gateway of 03-04, now declarative. Limitation: only at the edge; between internal services the Ingress plays no part.

(c) By weight between internal services. Sending 10% of the gateway's calls to orders-service to v2, with exact weight and regardless of how many replicas there are, is what a service mesh provides with a VirtualService (05-05, where we close this thread).

In any of the three, the hard part is not the YAML but the decision to move forward: comparing errors and latency of v2 with v1 for N minutes, with enough traffic for it to be significant (at 4 a.m., 10% of almost nothing says nothing). At first TechCorp will do it by hand looking at Grafana (06-01); the automation is in section 10.

The previous strategies control which code runs. A feature flag (04-03: PAYMENT_NEW_PROVIDER, REMOTE_CATALOG) controls which code executes out of what is running. With flags, the deployment of 1.1.0 can be done with the functionality turned off (dark launch): the new code reaches production with a normal rolling update, risk-free, and the release (turning it on for 1%, 10%, everybody) is a configuration change, in seconds, without deploying and without rollout undo. Compared with the canary:

Rule: a flag has an expiration date; when the functionality is the only one, the if is deleted.

All of them share the four requirements of section 1 and the expand/contract discipline.

With a rolling update of Orders, for a while there are v1 and v2 pods consuming the same queue orders.saga (04-04). RabbitMQ spreads the messages across all connected consumers, knowing nothing about versions. Consequences and rules:

• Both must understand the same events: the tolerance to new fields and the event versioning of 03-06 are not theory, they are what makes it possible for a stock.reserved published by the new Inventory to be processed by an old Orders.
• Redistribution when a pod dies: when v1-a receives SIGTERM halfway through processing a payment.confirmed, the unacknowledged message (ack pending) goes back to the queue and another pod receives it, maybe a v2. The idempotency of processOnce (04-04, processed_events table) is what prevents confirming the order twice. The consumer's graceful shutdown must stop taking messages (channel.cancel), finish the ones it has, and only then close; the low prefetch (10) bounds how many are left half-done.
• If v2 publishes a new event (order.confirmed with one more field), the consumers in other services at their current version must tolerate it: expand/contract applies to events too.
• Topology change (new queue, new binding): it is declared idempotently at startup (03-02) and before publishing to it; a binding the previous version still uses is never deleted.

A queue consumer cannot be made "purely" blue-green (green already consumes as soon as it connects): either green is deployed without starting the consumer (a CONSUMER_ACTIVE flag) or you accept that it consumes from the start, which is what TechCorp does because consumption is already idempotent.

• Rollback: kubectl rollout undo, selector change, withdrawing the canary or reverting the commit in platform. It is the option when the failure is obvious and the previous version is compatible with the current state (which is why migrations only add). Target: minutes, and it is what lowers the MTTR of 05-03.
• Forward-fix: deploying a new version that fixes things, instead of going back. Necessary when going back is impossible (a contract phase already applied, data already written in a new format) and acceptable when the pipeline takes less than 15 minutes: the better the CI/CD, the easier it is to justify the forward-fix.
• Progressive automation: Argo Rollouts (a Rollout resource that replaces the Deployment, with strategy: canary: steps: [setWeight: 10, pause: 10m, analysis: ...]) or Flagger (operates on the normal Deployment and creates the canary on its own) query Prometheus, advance the weight if the metrics comply and revert if not. Both need a weighting mechanism (NGINX Ingress, Istio, Linkerd) and the metrics of 06-01. It is the natural destination of the manual canary of Orders and Payments, once the Platform team has the mesh or the weighted Ingress and the metrics ready.

• Default maxUnavailable (25%) with two replicas: during the deployment one is left; and with replicas: 1 the rolling update is a recreate under another name. maxUnavailable: 0 and at least two replicas on any service that receives traffic.
• Destructive migration in the same deployment as the code: the Job renames the column, the v1 pods start failing before v2 is ready, and rollout undo fixes nothing. Expand/contract always.
• Canary without metrics or traffic: "we had it for 10 minutes and nothing happened" at 3 a.m. Define what is watched, for how long and how much minimum traffic before starting.
• Blue-green with state in the pod (sessions, local cache): green starts "cold". TechCorp's services are stateless precisely for this reason.
• Forgetting the readinessProbe or making it trivial (/health/live for both): Kubernetes deletes old pods as soon as the new ones "start", not when they can serve.
• Eternal flags: every if (flags.isActive(...)) is a branch to test. Removal date on the ticket that creates it.
• Tip: rehearse the rollback in staging as part of the pipeline (rollout undo + E2E) at least once per major version; a rollback that has never been tested is not a plan.

Exercise 1. orders-service has replicas: 2 and the default RollingUpdate configuration. Luis notices that during every deployment the p95 latency doubles and there are some 502s at the gateway. Explain the two most likely causes with what we have seen and write the spec fragment that fixes them.

Exercise 2. The Orders team wants to rename the unit_price column of order_lines to price (03-06 left it as an example of an incompatible change). Design the complete expand/contract sequence: migrations (numbered from 007-...), what the code writes and reads in each version (1.3.0, 1.4.0, 1.5.0) and at which point it is safe to apply each step.

Exercise 3. Marta asks why the gateway goes blue-green and not canary "like Orders, which is more modern". Write the answer in five lines, and add what would be needed for the gateway to go canary safely.

Solution 1. (1) With replicas: 2 and maxUnavailable: 25% (rounded to 1 pod), at some point there is a single pod serving all the traffic: latency doubles. (2) The 502s come from the window between "the pod leaves Endpoints" and "the pod stops accepting": if the readinessProbe takes 5 s to fail three times (15 s) but the process has already closed, or if the gateway keeps keep-alive connections to the dying pod; the graceful shutdown of 04-02 (immediate 503 on /health/ready and server.close()) narrows the window, and a preStop with sleep 5 covers it. Fix:

terminationGracePeriodSeconds: 30 is still enough: 5 s of preStop + less than 10 s of shutdown.

Solution 2. v1.3.0 (expand): 007-lines-price-expand.sql: ALTER TABLE order_lines ADD COLUMN price NUMERIC(10,2); (nullable); the code writes both columns and keeps reading unit_price. Compatible with v1.2.x (it ignores price, which allows NULL). Data migration (separate Job, or inside 007 if the table is small; in batches otherwise): UPDATE order_lines SET price = unit_price WHERE price IS NULL;. v1.4.0 (read switch): the code reads price and keeps writing both; ALTER COLUMN price SET NOT NULL can be added in 008-... once it is filled in. Compatible with v1.3.0 (which writes both). v1.5.0 (contract): 009-lines-price-contract.sql: ALTER TABLE order_lines DROP COLUMN unit_price;; the code no longer writes it. It is only safe when no replica older than v1.4.0 can come back (days later, after confirming there will be no rollout undo to v1.3.0). Three deployments instead of one, and each of them can be undone.

Solution 3. The gateway is the single entry point: any failure affects 100% of the requests of every service, and its typical change is routing or a library, hard to "test with 10%" because a configuration error is usually all or nothing, not statistical. Blue-green allows deploying the full green, running the E2E and route tests against gateway-green before it receives traffic, and going back to blue in a second with an Ingress change, without depending on metrics. To move to canary safely it would take: exact per-request weight (NGINX Ingress canary-weight, already seen), RED metrics per gateway route (06-01) with a defined error and latency threshold, enough traffic in the analysis window, and preferably the automation of Argo Rollouts/Flagger so as not to depend on someone watching Grafana. Until then, blue-green is the safest option for that piece.

Zero-downtime deployment rests on four pillars (readinessProbe, graceful shutdown within terminationGracePeriodSeconds, more than one replica and coexistence of versions) and materializes in four strategies: recreate only when coexistence is impossible; rolling as the default (maxSurge: 1, maxUnavailable: 0, minReadySeconds, rollout undo); blue-green with two Deployments (-blue/-green) and the Service's selector.version (or the Ingress) as the switch; and canary by replicas, by weight and X-Canary header in the NGINX Ingress, or by exact weight between internal services with a mesh. Coexistence imposes compatible contracts (03-06) and expand/contract migrations (the currency column of order_lines in two deployments), also for the event consumers that share a queue during the rolling update (idempotency of 04-04). Feature flags separate deployment from release; TechCorp's decision is rolling for Catalog, Inventory, Notifications and Customers, canary for Orders and Payments, blue-green for the gateway; and Argo Rollouts or Flagger will automate the canary once there are metrics and exact weights. That exact weight between internal services, together with timeouts, retries and mTLS that today each service would solve in its own code, is what a service mesh promises; the next lesson examines what Istio (and Linkerd) provides, what it costs, and whether TechCorp needs it yet.