We have the cineclick/churn-api:0.1.0 image running in a container. That's great progress, but a single container is not production: if the process dies in the middle of the night, it stays dead; if traffic multiplies during a premiere night, nobody adds capacity; if the new version turns out to be broken, nobody pulls it. All of that — restarting, replicating, scaling, updating without downtime — is the job of an orchestrator, and the industry standard is Kubernetes. In this lesson you'll learn the four Kubernetes concepts an ML engineer needs (Pod, Deployment, Service, HPA), we'll write the churn service's real manifests — with probes pointing at our /health, configuration via ConfigMap, and secrets via Secret —, we'll practice rolling out and rolling back a new image, and we'll evaluate the serverless alternative. This is not a Kubernetes course (the portal has a dedicated one if you want to go deeper); it's exactly the slice you need to deploy and operate your model.

1. What an orchestrator gives you
2. Minimal Kubernetes for ML engineers: Pod, Deployment, Service, HPA
3. The churn-api service manifests
4. Configuration and secrets: ConfigMap and Secret
5. Rollout and rollback: two distinct levels of "going back"
6. Autoscaling with the HPA
7. The serverless alternative: scale to zero and cold starts
8. Managed and specialized serving, in one brushstroke
9. CineClick's decision

An orchestrator turns "I have an image" into "I have an operated service". Its contributions, in the order you'd miss them:

• Replicas: it runs N identical copies of the container and spreads traffic among them. More capacity and, above all, fault tolerance: one fallen replica doesn't take the service down.
• Self-healing: it watches every container (with the probes we'll see) and automatically restarts any that dies or stops responding. "Who restarts it at 3 a.m.?" has an answer: the machine.
• Rollout and rollback: it deploys a new version by gradually replacing replicas without interrupting the service, and lets you return to the previous one with a single command.
• Service discovery and load balancing: it gives the set of replicas a stable DNS name and a single IP; consumers don't know (and shouldn't know) how many replicas there are or where they live.
• Declarative management: you describe the desired state in YAML ("I want 3 replicas of this image with these resources") and Kubernetes works continuously to make reality match. This fits the course's philosophy: just as dvc.yaml declares the pipeline, the manifests declare the deployment — everything in Git, everything auditable.

Four concepts are enough for our case:

The mental rule: you never create Pods by hand — you create a Deployment that creates and cares for the Pods, and a Service that gives them a front door. The Pod is cattle, not a pet: don't get individually attached, because Kubernetes will kill and recreate it whenever it suits (and that's good: it's why the model is downloaded from the registry at startup and depends on nothing local).

The manifests live in the repo, under deploy/k8s/. First the Deployment, the central file:

Points worth explaining:

• requests and limits: the requests are the reservation Kubernetes uses to decide which machine the Pod fits on; the limits, the ceiling. For an ML service, size memory by looking at how much the loaded model occupies (our 300-tree RandomForest takes a few hundred MB in memory; a 512Mi request with a 1Gi limit leaves room). A too-tight memory limit produces the classic Pod that dies OOMKilled right as it loads the model.
• Both probes point at /health, the endpoint we wrote in 04-02 — and now its design pays off: since /health verifies the model is in memory, the readinessProbe guarantees that a freshly created Pod receives no traffic until it has finished downloading the champion from the registry. The livenessProbe detects hung processes and restarts them. And here the "fail fast" circle from 04-02 closes: if the registry is down, the new Pod dies at startup, Kubernetes retries it with increasing backoff (CrashLoopBackOff), and meanwhile the 3 old replicas keep serving — the risk we accepted in 04-03, mitigated exactly as promised.
• replicas: 3: a reasonable production minimum — it survives the loss of one replica and rollout restarts without dropping to zero.

The Service, much shorter:

From any Pod in the cluster, the frontend calls http://churn-api/predict — a stable name, automatic balancing across the ready replicas. (Exposure outside the cluster — Ingress, the cloud's load balancer — depends on each company's platform, and we leave it noted.)

Apply and verify:

The principle from 04-03 — same image for every environment, configuration comes in from outside — materializes in two objects:

• The production ConfigMap points at the cluster's MLflow; staging's, at another. The image doesn't change.
• The Secret holds the API key from 04-02. Careful: the example YAML is illustrative — in a real repo, Secrets are not committed in the clear; they're created with kubectl create secret, or with platform tooling (Sealed Secrets, External Secrets, the cloud's secret manager). The 04-03 rule holds one level up: secrets live neither in the image nor in Git.

Suppose we publish version 0.2.0 of the service (the new /predict-batch endpoint from 04-02's exercise). Deploying means changing the Deployment's image:

Kubernetes performs a rolling update: it creates Pods with the new image, waits for their readinessProbe to pass (model downloaded, /health OK), sends them traffic, and only then retires old Pods — one by one, without service interruption. If 0.2.0 turns out to be broken:

And here it's worth stopping, because CineClick now has two levels of rollback that must not be confused:

This separation is a direct fruit of the 04-03 decision (model outside the image): each problem gets reverted with the tool of its own level, without dragging the other along. If the model were baked into the image, both rollbacks would be the same one and you'd always pay full price. Final note: the rolling update is Kubernetes' standard strategy; the advanced release strategies — shadow, canary, A/B, which decide how much traffic the new version sees before it's trusted — are the subject of lesson 05-04.

With 3 fixed replicas, a premiere night with a cancellation spike saturates the service, and a Tuesday's small hours waste money. The HPA adjusts the replicas to demand:

How it reasons: the goal is to keep average CPU usage at 60% of the request (here, 60% of 250m). If the replicas run at 90%, the HPA computes how many it would take to return to 60% and creates them; at 20%, it removes some (never below 3). Two nuances for ML services:

• The HPA only works well if the requests are set right: they're the measuring stick for the percentage.
• CPU is a good proxy for our inference (the RandomForest's predict is pure CPU), but remember that each new replica takes as long to become ready as it takes to download the model — the HPA reacts to spikes in tens of seconds, not instantly. For predictable spikes (a premiere campaign), scaling preemptively (kubectl scale --replicas=6) remains legitimate.

Kubernetes requires having (or paying for) a cluster and someone who understands it. The serverless containers alternative (Cloud Run, Azure Container Apps, AWS App Runner / Lambda with images) takes your very same Docker image and handles everything: provisioning, request-based scaling, pay-per-use billing... including scaling to zero when there's no traffic.

The price of scaling to zero is the cold start: the first request after a quiet period must wait for a container to spin up — and in our case startup includes downloading the model from the registry. For churn-api that would be several seconds; for a multi-gigabyte deep learning model, tens. A 5 s cold start against a 300 ms budget is a 16x violation.

A note we'll pick up in 04-05: CineClick's batch pattern, which runs 10 minutes a week, is an ideal candidate for ephemeral/serverless compute — paying for a job that runs, not for a server that waits.

There is a third path you should know about even though we won't use it: ML-specific serving platforms.

• KServe and Seldon Core: layers on top of Kubernetes that understand models — you hand them a model URI (they support MLflow) and they generate the service, with extras like request-based autoscaling, native canary, or explainers. In exchange, another platform piece to operate.
• Managed cloud endpoints (SageMaker Endpoints, Vertex AI Prediction, Azure ML Endpoints): the provider operates everything; you upload the model. Maximum convenience, in exchange for vendor coupling, cost, and less control over the container (our requirement of importing build_features from the package demands a custom container or custom images).

They're reasonable options in organizations already married to a cloud or with dozens of models to serve. For a first service, understanding it to the bottom — as we're doing — is worth more than delegating it to a black box.

Let's apply the criteria: the cancellation flow has sustained traffic throughout the day (people cancel at any hour), a strict 300 ms budget that rules out cold starts, and CineClick's platform already operates a managed Kubernetes cluster in its cloud for the rest of its microservices — there's a team, monitoring, and habit. The decision, documented next to 04-01's:

The churn-api service is deployed on CineClick's managed Kubernetes in its cloud: a Deployment with 3 replicas (image cineclick/churn-api:0.1.0), Service churn-api, probes against /health, configuration via ConfigMap (MLFLOW_TRACKING_URI, CHURN_THRESHOLD) and the API key in a Secret, HPA from 3 to 10 replicas at 60% CPU. Serverless is ruled out for this service because of cold starts against the 300 ms budget, but noted as an option for the weekly batch job.

The manifests stay in deploy/k8s/ inside the repo; today we apply them with kubectl apply by hand, and that phrase — by hand — should already make you uncomfortable. It's the gap module 5 comes to fill.

• Mistake: deploying without a readinessProbe (or with one that doesn't check the model). Kubernetes would send traffic to Pods still downloading the model from the registry: 500 errors on every rollout and every scale-out. Our probe against /health (which verifies the model in memory) prevents it — 04-02's design at work.
• Mistake: memory limits below what the loaded model occupies. The unmistakable symptom: Pods going OOMKilled/CrashLoopBackOff right after starting. Measure the process's memory with the model loaded and leave headroom.
• Mistake: confusing the two rollbacks. "The new model is doing badly" is not fixed with kubectl rollout undo (that reverts the image): it's fixed by moving @champion in the registry and restarting the Pods. And vice versa.
• Mistake: an overly aggressive liveness. A short initialDelaySeconds plus a slow registry = Kubernetes killing Pods that were about to be ready, in a loop. Give the startup room (ours: 30 s) and leave the fine-grained demands to the readiness probe.
• Mistake: treating the HPA as magic. Without realistic requests it doesn't scale well, and it's not instantaneous: every new replica pays the full startup (container + model download). For known spikes, scale preemptively.
• Tip: the manifests live in the repo (deploy/k8s/), versioned like the code and the pipeline. In 05-02 it will be CD that applies them; having them in Git already is half the battle.
• Tip: if you want to go deeper into Kubernetes (Ingress, namespaces, RBAC, operators), the portal has a dedicated course; here we deliberately stay within the slice an ML engineer operates daily.

Marketing launches a TV campaign on Saturday at 21:00 and visits (and cancellations) are expected to triple between 21:00 and 23:00. With the HPA configured (3–10 replicas, 60% CPU), what would you do on Friday? Reason out why the HPA alone might not be enough and give the specific command.

After deploying image 0.2.0, the logs show /predict-batch returning 500 errors due to a serialization bug, even though /predict works. In parallel, Laura suspects model v2 is degrading and wants to go back to v1. State the two actions, with their commands/steps, and explain why they are independent.

A new Pod has been in CrashLoopBackOff for 10 minutes. kubectl logs on the Pod shows: KeyError: 'MLFLOW_TRACKING_URI'. Diagnose the most likely cause and the steps to confirm and fix it. Why do the other replicas keep working?

Solution 1

The HPA reacts after the CPU rises, and each new replica takes time to become ready (schedule the Pod + start the container + download the model + pass the readinessProbe): in a vertical spike at 21:00, the first minutes would be served with insufficient capacity. For a predictable spike, scale preemptively: on Friday (or Saturday afternoon), kubectl scale deployment/churn-api --replicas=8. An important nuance: with the HPA active, this manual scaling coexists with it — the HPA won't go below its own calculation, but it could scale down if CPU is low before the spike; the robust option is to temporarily raise minReplicas to 8 in the HPA (editing hpa.yaml and applying it) and restore it on Sunday. The essence: reactive autoscaling for the unpredictable, planning for the announced.

Solution 2

• Image 0.2.0 bug: kubectl rollout undo deployment/churn-api — an inverse rolling update back to 0.1.0, with no service interruption. Afterwards, fix the bug, publish 0.2.1, and redeploy.
• Model v2 degradation: that's the 03-02 flow, not Kubernetes' — with the evidence in hand and human review, move the @champion alias from v2 to v1 in the MLflow registry, then kubectl rollout restart deployment/churn-api so the Pods reload the model in their lifespan (verifiable at GET /version, which will start answering "1").

They are independent because in 04-03 we deliberately separated the life cycles: the image governs the code; the registry governs the model. You can revert either without touching the other — and in this scenario you do both, each with its own tool and its own decision process.

Solution 3

The most likely cause: the Pod starts without the MLFLOW_TRACKING_URI variable — remember that in 04-02 we read it with os.environ[...] precisely to fail fast if it's missing. Since the variable comes from a ConfigMap, the suspects are: the churn-api-config ConfigMap doesn't exist in that namespace, the key is named differently (mlflow_tracking_uri), or the Deployment references the wrong name. Confirmation: kubectl describe pod <pod> (it will show ConfigMap reference errors) and kubectl get configmap churn-api-config -o yaml (verify existence and key). Fix: create/correct the ConfigMap and let the Deployment recreate the Pod (or kubectl rollout restart). The other replicas keep working if they started before the problem (for example, someone deleted or renamed the ConfigMap afterwards): variables are injected when the Pod is created, so the living ones keep their value — another example of why "it works on the old Pods" doesn't prove the deployment is healthy.

The churn service is now a production citizen: 3 replicas (up to 10 with the HPA) of the cineclick/churn-api:0.1.0 container behind the churn-api Service, with Kubernetes restarting whatever dies, holding traffic back until each Pod has loaded the champion (readinessProbe against /health), configuration entering via ConfigMap and secrets via Secret, and two well-separated rollback levers: kubectl rollout undo for the image, moving @champion for the model. You also know when this machinery is overkill — sporadic traffic and tolerance for cold starts: serverless — and that specialized serving platforms exist for when the models number in the dozens. One uncomfortable question remains before closing the module: is this service fast? How long does a prediction really take, where does the time go, how much does every thousand calls cost, and what can be gained before paying for more infrastructure? The next lesson measures first — percentiles, not averages — and optimizes afterwards, in order of return.