The previous lesson's CI ends at a green tick, but a green tick doesn't serve requests: someone is still building the image on their laptop, pushing it to the registry, and carefully typing kubectl apply. Continuous delivery (CD) automates that final stretch — from validated code to the cluster — and in ML it has a peculiarity worth understanding before writing a single line of YAML: at CineClick there isn't one thing to deploy, there are two, and they travel by different roads. The service (code and Docker image) changes when Marc touches the API; the model (an artifact in the MLflow registry) changes when Laura trains a better challenger. This lesson builds both pipelines: the service's with a GitHub Actions workflow triggered by version tags, and the model's as an alias-promotion flow with a human in the loop.

1. The key distinction: two delivery pipelines, not one
2. CD for the service: .github/workflows/cd-service.yml
3. CD for the model: moving @champion builds nothing
4. Automatic gates vs. human approval
5. Secret management in CD
6. Rollback in both pipelines
7. The two pipelines, together, in one diagram

Recall the decision from module 4 (lesson 04-03): the cineclick/churn-api:0.1.0 image does not contain the model. At startup, the container downloads models:/churn-cineclick@champion from the registry using MLFLOW_TRACKING_URI. That decision, which back then looked like a packaging detail, is what now makes it possible to decouple the two deliveries:

Why does it matter so much not to mix them? Because it couples cadences that have nothing to do with each other. If the model lived inside the image, every new model would force building, scanning, publishing, and deploying an image — and every API release would "drag along" whatever model was current, making it hard to answer the most basic operational question: what changed, the code or the model? Decoupled, each pipeline has its own trigger, its own validation, and — crucially — its own rollback (the two levels we already separated in 04-04).

CI runs on every push; deployment must not. The convention we adopt: deploy only when a semantic tag (v0.2.0) is created. Creating a tag is a deliberate act — "this is a release" — which also leaves the deployment history written in Git:

• on: push: tags: — the v*.*.* pattern matches v0.2.0 but not working tags. GITHUB_REF_NAME holds the tag name; we strip the v to tag the image as 0.2.0.
• Tests again — wasn't that CI's job? — yes, and the fast ones still get repeated: a tag can be created on a commit that never went through a PR. Thirty seconds of pytest is cheap insurance against deploying something that was never validated.
• Double tag: version + SHA — 0.2.0 is the "human" tag; ${{ github.sha }} identifies the exact commit immutably. If someone ever re-tags 0.2.0 by mistake (they shouldn't, but it happens), the SHA doesn't lie. Deploying by SHA is the paranoid-correct option; we publish both.
• Multi-stage build — the same two-stage Dockerfile from module 4 (builder with pip-tools, ~380 MB runtime). CD doesn't change how it's built, it changes who builds it: an ephemeral, audited runner instead of whoever's laptop.
• environment: staging / environment: production — GitHub environments group secrets per environment and, in production, have required reviewers configured: the job pauses until an authorized person (Laura or the platform lead) clicks "Approve". It's manual approval inside the pipeline, with who-and-when recorded.
• Smoke test — before asking anyone to approve, the pipeline checks in staging that /health responds and that /version returns exactly the freshly deployed version. A smoke test doesn't validate quality; it validates that what we deployed is what we think it is and it starts. Without it, the human approval would be approving blind.
• kubectl rollout status --timeout — turns "I applied the change" into "the change is healthy or the job fails". Without this line, the pipeline can stay green with the pods in CrashLoopBackOff.

One deliberate limitation must be said out loud: after approval, production receives the new version all at once, across all 3-10 replicas (a standard rolling update). For API code that's usually acceptable; for a new model it's riskier than it looks, because offline metrics don't guarantee behavior on real traffic. Lesson 05-04 replaces this "all at once" with shadow, canary, and A/B.

Here comes the lesson's mental reframing moment: deploying a new model builds no image at all. The "deployment" is an operation on registry metadata — moving the @champion alias to another version — followed by a service reload. The full flow:

Step by step:

1. Validated challenger. The candidate (say, the v3 Laura registered as @challenger) has already passed the 05-01 model tests: smoke, invariance, directionality, and thresholds (recall ≥ 0.60, precision ≥ 0.50). In addition, before proposing promotion it's evaluated on relevant subgroups: does it hold its recall on customers with less than 6 months of tenure? And per plan? A model can improve the global average while degrading exactly the segment retention cares about most.
2. Documented human approval. Consistent with module 3's policy: promotion ALWAYS goes through a person. In practice: a promotion PR or issue with the metrics report (global and per subgroup), the comparison against the current champion, and explicit approval from Laura as the model's owner. The decision stays written and linkable — a year from now someone will ask why v3 was promoted, and there will be an answer.
3. Move the alias. The operation itself is minimal, which is exactly why it should be done with a versioned script instead of clicks in the UI:

4. Reload the service. The pods loaded the model at startup, so they keep serving the old one until they restart. The standard reload is a progressive restart — no downtime, because the /health probes guarantee no new pod receives traffic until it has the model loaded:

5. Verify. Module 4's /version endpoint returns the loaded model's version; the check is a curl and an eyeball:

Could this whole flow be automated in another Actions workflow? Steps 3-5, yes (and mature teams do it, triggered by the promotion PR's approval). What does not get automated is step 2 — and not out of nostalgia, as the next section justifies.

The general pattern: automate the execution, not necessarily the decision. Automatic gates filter (nothing below the bar reaches a person); the person decides on what the tests can't see. At CineClick this is also consistency with what was agreed in module 3: the criterion "precision ≥ 0.50 if recall > 0.60" is a necessary condition for promotion, not a sufficient one.

The CD pipeline touches sensitive credentials: cluster access (KUBECONFIG_STAGING, KUBECONFIG_PROD), the image registry, the DVC remote, and the MLflow server. Non-negotiable rules:

• Never in the repo: not in the YAML, not in configs/, not "temporarily" in a commit. Git history doesn't forget.
• Never in the image: module 4's image was already built without secrets; the model is downloaded at runtime with credentials injected by Kubernetes (the churn-api-secrets Secret). CD doesn't change this — the runner uses its secrets to deploy, and the pod uses its own to operate. They are different credentials with different permissions (the pod can't deploy; the runner doesn't serve predictions).
• GitHub Secrets per environment: KUBECONFIG_PROD lives in the production environment, so only jobs that passed the approval can read it. A job from a random branch can't even see it.
• Least privilege: the pipeline's kubeconfig points to a ServiceAccount that can only touch the churn-api deployment in its namespace, not the whole cluster.
• Rotation: if a secret may have leaked (a log that printed it, a lost laptop), rotate it and move on. GitHub masks secrets in logs, but it doesn't detect transformations (base64, for example).

The two rollback levels we separated in 04-04 now map exactly onto the two pipelines — and that symmetry is the reason we insisted on decoupling them:

Two operational tips: first, rollbacks get rehearsed — a rollback that has never been executed is a hypothesis, not a plan; at CineClick it's tried in staging after every release. Second, thanks to the workflow's double version+SHA tag, there's always an exact previous image to go back to; already-published image tags are never overwritten.

Two roads, two triggers, two rollbacks — one cluster where they converge. The image says how predictions get made; the alias says with what.

• Baking the model into the image "to keep things simple". You've just merged the two pipelines: every new model requires an image release, and every image release freezes a model. You also lose the cheap alias rollback. Module 4's download-at-startup exists for exactly this.
• Deploying from main on every merge, without tags. It works until an innocent merge (a README) redeploys production on a Friday. The tag makes the release intentional and leaves a version inventory in Git.
• A smoke test that only checks for HTTP 200. /health can answer OK with the old version if the rollout never got around to replacing the pods. Always check the content of /version against the expected version — it's the difference between "something responds" and "what I deployed responds".
• Manual approval without information. An "Approve" button without the metrics report in front of it is compliance theater. The human gate is worth exactly the context presented to it: link the subgroup report in the promotion PR.
• Forgetting the rollout restart after moving the alias. The alias points to v3 but the pods keep serving the v2 loaded in memory — and /version would have told you. It's the most common mistake in the whole model flow; that's why the script reminds you when it finishes.
• Sharing credentials between the pipeline and the pod. If the pod's token can deploy, a compromised container can redeploy the cluster. Separate credentials, minimal permissions.
• Tip: write the model promotion procedure (steps 1-5) into the repo, in docs/model-promotion.md, and have the promotion PR use a template with a checklist. When something is both infrequent and critical, human memory is the worst possible storage.

Marc asks: "If the model gets downloaded from the registry at startup, why don't we have the pod check every 5 minutes whether @champion changed and hot-reload it, without a rollout restart?". Give one argument for and two against automatic hot reloading, thinking about operations (hints: auditability, replicas, memory).

The v0.3.0 tag has just been deployed and ten minutes later the 500 error rate spikes. /version shows service_version: 0.3.0, model_version: 2. The last model change was three weeks ago. Which rollback do you execute, with which exact command, and what do you check afterwards? Why do you rule out the other rollback?

Write the fragment of the deploy-production job that would be missing to run a smoke test in production too, after the rollout (URL https://churn.cineclick.internal), checking /health and that service_version matches the deployed version. What should happen if the smoke test fails in production?

Solution 1

In favor: model promotion would complete without touching Kubernetes — fewer steps, fewer chances to forget the restart. Against: (1) auditability and control: the model change would stop being an explicit, verifiable event (a rollout with its history) and become a process that happens "sometime in the next 5 minutes" on each pod, hard to correlate with incidents; (2) inconsistency between replicas: during the reload window, some replicas would serve v2 and others v3 depending on their timer, and two identical consecutive requests could receive different predictions with no deployment to explain it; besides, hot reloading temporarily duplicates the model in memory (old and new coexist during the swap), which can push the pod against its memory limit at the worst possible moment. The progressive rollout restart gives you the same thing with guarantees: fresh pods, probes that verify the load, and an auditable event.

Solution 2

Service rollback: the symptom arrived with the v0.3.0 release and the model hasn't changed in three weeks (model_version: 2 confirms it). Command:

Follow-up check: curl .../version must show service_version: 0.2.0 (the previous one) and the 500 rate must return to normal. The model rollback is ruled out because there's no hint that the model is the problem — moving @champion wouldn't fix a code bug and would add a second variable to the incident, complicating diagnosis. Rule of thumb: revert the last thing that changed, and only one thing at a time.

Solution 3

If it fails, the job goes red and the team must treat it as an incident: the deployment already happened (this is post-deployment verification, not a prior gate), so the action is exercise 2's service rollback. Some teams add the kubectl rollout undo as an automatic step with if: failure() — a reasonable option here because the smoke test is objective; we'll explore it with more nuance in 05-04, where automatic rollback is based on real-traffic metrics and not just an endpoint.

CineClick now delivers hands-free, and it does so along two well-separated roads. The service travels through cd-service.yml: a v*.*.* tag triggers tests, a multi-stage build, a push to ghcr.io/cineclick/churn-api with a double tag (version and SHA), deployment to staging, a smoke test against /health and /version, approval by a required reviewer in the production environment, and a rollout to production. The model travels through the registry: automatic gates (the 05-01 tests plus subgroup validation), documented human approval — module 3's policy, intact —, scripts/promote_champion.py to move the alias, a rollout restart, and verification at /version. Each road with its own rollback: rollout undo for the image, alias back for the model. The secrets live in per-environment GitHub Secrets and in the cluster's Secret, never in the repo or the image. Two loose ends remain, both announced: the production deployment is still "all at once" after staging — 05-04 will turn it into a gradual process that earns trust with real traffic — and there's an entire process that fits no GitHub Actions workflow: the Monday 06:00 batch scoring, which no commit triggers but the calendar does. That takes another piece — an orchestrator — and it's exactly the next lesson.