The four previous lessons compared CI/CD tools with one another. This one does not, because Docker and Kubernetes are not CI/CD tools: they are the substrate. Docker shows up twice in any modern pipeline and in completely different roles — it is the environment the pipeline runs in and it is the format of the artifact you deploy — and confusing those two roles produces 1.2 GB production images containing the compiler, the tests and the build credentials. Kubernetes, for its part, is the most common destination today and it changes the shape of CD: the push model Reservalia uses against ECS has a natural equivalent in Kubernetes, but the community moved largely towards GitOps, which inverts who starts the deployment. The course has been using both since module 2 without going into detail; this lesson goes into detail. And it ends with the question almost nobody asks out loud: when you do NOT need Kubernetes.

1. The two roles of Docker in a pipeline
2. BuildKit: mount caches, multi-stage and secrets
3. Layer cache in the registry and multi-platform builds
4. Minimal images, non-root, HEALTHCHECK and OCI labels
5. Why the digest beats the tag
6. Building images inside the pipeline: DinD, socket and daemonless builders
7. Kubernetes as a destination: the minimum set of objects
8. Probes and their role in the rolling update
9. Rollback: kubectl rollout status and undo
10. Parameterising per environment: Helm and Kustomize
11. Push versus pull: why GitOps won in Kubernetes
12. Reservalia on Kubernetes, contrasted with ECS
13. Kubernetes as a platform for the runners themselves
14. When you do NOT need Kubernetes
15. Common Mistakes and Tips
16. Exercises
17. Conclusion

Role A is what gives reproducibility to the pipeline: a job running on node:22.11-bookworm behaves the same today as it will in six months' time, regardless of what is installed on the machine executing it. It is the answer to the contaminated agent from 06-01, and that is why every modern tool uses it by default.

Role B is the immutable artifact from 02-06. Confusing the two produces the most common anti-pattern in the whole topic: using the same image to build and to run. The result is a production image that includes the TypeScript compiler, devDependencies, the Git history and, if you are lucky, no build credential file. The separation is done with multi-stage, which the course introduced in 02-03 and which we take into detail here.

BuildKit is the modern build engine (the default in recent Docker versions and in buildx). It brings three things that genuinely change a pipeline: parallel building of the stage graph, mount caches and secrets that do not end up in the image.

1. The # syntax= line is not a comment: it tells BuildKit which frontend version to use, and it is what enables --mount. Without it, those lines fail.
2. --mount=type=cache is the most profitable improvement and the least well known. The npm cache is mounted during the RUN and does not end up in the image, but it persists across builds on the builder. The difference from the classic "copy only package.json first to take advantage of the layer cache" trick: this one works even when the lockfile changes, because it does not depend on invalidating a layer but on having the tarballs already downloaded. sharing=locked avoids corruption with concurrent builds.
3. npm ci --omit=dev after building is what stops devDependencies reaching the runtime. Saving hundreds of megabytes this way is routine.
4. A separate, minimal runtime image: section 4.
5. USER nonroot: section 4.
6. OCI labels: section 4.

And build-time secrets, which is where most people get it wrong:

The rule is the one from 04-03 with a concrete mechanism: ARG and ENV holding secrets are visible via docker history to anybody who has the image. Deleting the file in a later RUN does not help: layers are immutable and cumulative. --mount=type=secret is the only correct way.

On an ephemeral runner there is no local layer cache: every build starts from scratch. The solution is to store the cache in the registry, which is what Reservalia's ci.yml was already doing and which we explain here.

Two warnings. The cache in the registry also has to be cleaned up: it is one more tag that grows, and with mode=max it grows a fair bit. And a poisoned layer cache is a real vector: if somebody with write access to the registry replaces the cache, your builds can pick up someone else's layers. Write permissions on the cache repository must be the same as on the images themselves (04-03).

Multi-platform with buildx, relevant ever since ARM runners became cheaper and development laptops became ARM:

This produces a manifest list: a single tag pointing to two images, and the destination downloads the one matching its architecture. Honest trade-offs: if the runner is amd64, the arm64 variant is built through QEMU emulation and can take between three and ten times longer; the alternative is to build each architecture on a native runner and join the manifests with docker buildx imagetools create, which is faster and more complex. And if your application has dependencies with native binaries, you have to test properly on both architectures: compiling is not the same as working.

The difference that matters is not the size, it is what is inside. A distroless image has no shell, no package manager, no curl, no utilities: if somebody gets execution inside the container, they have nothing to work with. It also drastically reduces the noise from vulnerability scanners: most of the CVEs that show up in a full Debian image are in packages your application never uses, and that noise is what makes teams stop reading the reports from 04-03.

A real trade-off, and it has to be said: debugging on distroless is awkward. You cannot do kubectl exec -it ... -- sh because there is no sh. The answers: ephemeral debug containers (kubectl debug --image=busybox --target=api) that attach to the Pod without modifying the image, and good observability (03-06) so you do not need to go in. Many teams use -slim in staging and distroless in production, which is a defensible compromise even though it slightly breaks the "the same artifact in every environment" rule.

Non-root user: by default a container runs as root, and although container isolation limits the damage, a root process that escapes has a lot more room. In Kubernetes this is reinforced with the Pod policy:

HEALTHCHECK in the Dockerfile is useful with Docker or Compose; in Kubernetes the orchestrator ignores it, using its own probes instead (section 8). Adding it does no harm — it documents how you check the process is healthy — but do not rely on it as a mechanism inside a cluster.

OCI labels: these are standard metadata that make the image traceable. org.opencontainers.image.source connects the image with its repository and revision with the exact commit. The question "which commit did what is in production come from?" — the first question in any incident, according to 03-05 — is answered with docker inspect instead of with archaeology.

This was already established in 02-06; here is the mechanism:

Practical consequences worth internalising:

• FROM node:22 in your Dockerfile is not reproducible. That tag points to different images over time. For genuinely reproducible builds: FROM node:22.11-bookworm@sha256:.... Trade-off: you have to keep it updated, and that is where the dependency automation from 04-02 comes in, which also refreshes base image digests.
• Deploying by tag introduces a race: between the pipeline building :v1.4.0 and the orchestrator pulling it, that tag can point to something else. That is why Reservalia's cd.yml deploys by digest.
• Promotion between environments means moving a digest, not rebuilding. Staging and production run exactly the same sha256:, and that is what makes "it worked in staging" mean something.
• Registries support immutable tags (ECR: tag immutability), which prevents overwriting an existing tag. Turn it on: it closes the door on a human error with serious consequences.

Building an image from a job that is already running inside a container is a real problem with three solutions and three very different risk profiles. It is one of the most important security decisions in a pipeline and it is nearly always taken out of inertia.

The reasoning behind why this matters: on a shared runner, a job with the host socket mounted can list and manipulate other jobs' containers, including those with production credentials in their environment. You do not need an external attacker: a modified .gitlab-ci.yml or workflow in a PR is enough. It is the most accessible privilege escalation in a badly configured CI.

Practical recommendation: if the runner is shared, do not use privileged DinD and do not mount the host socket. Kaniko or rootless BuildKit cover the normal case. DinD is acceptable on single-use ephemeral runners belonging to a single team, where the blast radius is the job itself. And on GitHub Actions with hosted runners the problem does not arise in the same way, because the machine is ephemeral and single-job — which is, incidentally, a design advantage of hosted runners that is not always appreciated.

You do not need to know the whole of Kubernetes to deploy to it. These are the objects you do have to understand:

1. maxUnavailable: 0 with maxSurge: 1 is the right setting for not losing capacity during the deployment: the new Pod is created first, you wait for it to be ready and only then is an old one retired. With maxUnavailable: 1 (the default value spreads things differently) you can end up below the capacity you need at peak time.
2. Image by digest, for the reasons given in section 5. On top of that, with a digest you do not need imagePullPolicy: Always: the identifier is already unique.
3. requests and limits are not optional. Without requests, the scheduler does not know what to reserve and places things badly; without limits, a Pod with a memory leak can take down the node. And one detail that bites: exceeding the memory limit kills the container (OOMKilled) with no warning, whereas exceeding the CPU one merely slows it down.
4. Probes: section 8.
5. preStop with a short pause solves a real race: when a Pod enters termination, Kubernetes sends it SIGTERM at the same time as it removes it from the Service endpoints, and those two things are not synchronised. Without the pause, some requests reach a Pod that is already shutting down. Five seconds eliminate most of the 502s during deployments.

The three probes are constantly confused and their effects are very different:

The readiness probe is the one that governs the deployment and the one that makes the rolling update from 03-04 safe: until the new Pod answers 200 on /health/ready, Kubernetes does not retire any old Pod. If the application starts up broken, the rollout stops with the previous version still serving: the failure contains itself.

Two configuration mistakes with serious consequences:

The liveness probe that checks dependencies. If /health/alive queries the database, a database outage makes the probe fail on every Pod, Kubernetes restarts them all at once, and on top of a database incident you get a total application outage with restart loops. Rule: liveness = "the process responds"; readiness = "I can serve requests, including my dependencies".

A liveness probe with no startup probe in slow applications. If the application takes 40 s to start and the liveness probe begins at 10 s with a threshold of 3 failures, the container restarts before it has finished starting, forever. The startupProbe exists for exactly this.

kubectl rollout status with --timeout is the line that turns a deployment into a verification: if the new Pods never become ready, the command returns an error and the CD job fails, instead of going green and leaving a stuck rollout that nobody looks at. It is exactly what 03-02 asked of an idempotent, verified deployment.

And rollback here is faster than on ECS: kubectl rollout undo restores the previous ReplicaSet, whose images are already downloaded on the nodes, so within seconds there are old Pods serving. Reservalia's 4-minute rollback by digest (03-05) would come down considerably.

Two honest caveats. undo reverts the Pod template, not the state of the world: if the deployment included a database migration, reverting the image does not revert the migration, and expand and contract still rules there (04-06). And --record is deprecated; in a GitOps flow the real history lives in Git, which is a better place than the object's annotations.

The same Deployment has to exist in dev, staging and production with a different number of replicas, different resources and different configuration. Two dominant approaches.

Helm — templates with variables:

--atomic --wait is the combination that turns helm upgrade into a deployment with automatic rollback built in: if the Pods do not become ready within the deadline, Helm undoes the change. It is the most valuable line in the command.

Kustomize — overlays on top of a base, with no templates:

In practice, many teams use both: Helm to install third-party software (ingress, monitoring, operators) and Kustomize for their own applications. It is a defensible and very common combination. The anti-pattern to avoid is an in-house chart with so many conditionals that you have to read Go templates to know what gets deployed.

Reservalia's cd.yml uses push: the pipeline holds the environment's credentials and performs the deployment.

The two reasons GitOps won in Kubernetes, and neither is fashion:

1. A cluster's administrator credentials are too powerful to leave in the CI. kubectl apply requires broad permissions; if the CI has them, compromising the CI is compromising the cluster. With GitOps the pipeline only needs permission to commit to a repository, which is a far smaller privilege.

2. Kubernetes is declarative, and that fits naturally with Git. The desired state is a set of manifests; putting them in Git and having an agent reconcile them continuously is the obvious extension. And it brings a capability push does not have: drift detection. If somebody runs kubectl edit at three in the morning during an incident, the agent detects it and either reverts it or flags it. "Why is production not what the repository says?" stops existing.

Honest trade-offs of GitOps: there is one more component to operate and update; the deployment stops being immediate unless you configure a webhook; the trail of a change crosses two repositories, which makes it harder to answer "which application commit is in production?" unless you automate it well; and it does not solve deployment strategies on its own — for canary or blue-green you need Argo Rollouts or Flagger, which is yet another piece. Lesson 05-03 already introduced it in the context of microservices; here you get the why.

An exercise in contrast, not in migration: Reservalia works well on ECS and 06-07 will explain why that matters.

What Reservalia would gain: near-instant rollback, portability, an enormous ecosystem of operators, better progressive delivery tooling, and GitOps with the reduction in CI privileges that comes with it.

What it would pay: a cluster that has to be upgraded several times a year, add-ons that have to be maintained (ingress, certificate manager, scaling, observability), operating Argo CD, and a real learning curve for a team of three. Diego would ask how much it costs and he would be right: with 340 businesses and ~9,000 appointments a month, the honest answer is that the 4-minute rollback they already have is not the bottleneck for anything.

When the answer would change: if the five services from 05-03 appeared with independent teams, if there were a multi-cloud requirement, or if the platform team grew enough that operating the cluster did not come out of product time.

One use that is sometimes forgotten: the cluster can host the CI agents, not just the application. It already came up with the Jenkins Kubernetes plugin (06-01) and with GitLab's kubernetes executor (06-02); in GitHub Actions it is ARC (Actions Runner Controller), which 06-06 covers in detail.

What it gives you: a clean environment per job — goodbye contaminated agent — free autoscaling (the cluster does it), use of idle capacity, and isolation by namespace between teams.

What it costs, honestly: a Pod takes tens of seconds to start and pull images, which makes queue time worse if you are not careful (images pre-pulled onto the nodes, or a small pool of warm Pods); the cache no longer persists by definition, so you have to lean on remote caches; and it is still a cluster to operate. Never run CI agents in the same cluster as production without strong isolation: a job is arbitrary code, and sharing nodes with production turns any escape into a serious incident.

This section is as important as the previous ones and it is left out far too often.

You probably do not need it if:

• You have fewer than five services. Kubernetes solves the coordination of many services; with few, what it mainly adds is complexity. Reservalia is the example.
• Your team has nobody who knows how to operate it. A badly operated cluster is worse than well-operated ECS or virtual machines. And "knows how to use kubectl" is not the same as "knows how to operate a cluster": upgrading versions, managing CRDs, diagnosing networking and storage, reviewing security policies.
• Your load is predictable and modest. Kubernetes' fine-grained autoscaling shines with spikes; with steady load, a managed service does the same with fewer pieces.
• You already have a managed service that works (ECS, Cloud Run, App Runner, App Service). Migrating costs months and you have to be able to name the specific problem it solves.
• The main reason is "portability between clouds". It is real but usually theoretical: the cluster is portable, but the managed database, the load balancer, the object storage, identity management and the cloud-specific add-ons are not. The portability you get is partial and you pay dearly for it.

You do need it when: you have many services and several teams that need autonomy (05-03); you need sophisticated progressive delivery and capabilities that only exist in that ecosystem; your load is very variable and efficient container bin-packing genuinely saves money; you have real, budgeted on-premise or multi-cloud requirements; or you already have the platform and the team, in which case the marginal cost of one more application is low.

The rule, which is the same one that closed module 5: complexity is justified by a specific problem it solves, not by the fact that other people use it.

Using the same image to build and to run. It produces enormous images with compilers and credentials. Multi-stage, always.

Secrets via ARG or ENV. They stay in the image history even if you delete the file afterwards. --mount=type=secret.

FROM node:22 without pinning. The build is not reproducible. Pin the version and, in sensitive environments, the digest, with automation to keep it updated.

mode=min in the registry cache with multi-stage. It loses the intermediate stages, which are precisely the expensive ones. mode=max.

Mounting the host's Docker socket in CI jobs. It amounts to handing root on the host to any PR. Kaniko or rootless BuildKit.

A liveness probe that checks the database. It turns a database incident into a total outage with restart loops. Liveness = process; readiness = dependencies.

No startupProbe in slow applications. A permanent restart loop that looks like an application bug.

No requests or limits. Bad scheduling, and a memory leak takes down the node. Remember that exceeding the memory limit kills the container with no warning.

Deploying by tag instead of by digest. It introduces a race and breaks the "the same artifact in every environment" guarantee.

kubectl apply without kubectl rollout status --timeout. The job goes green with the rollout stuck. False green (04-04) in its most expensive form.

Treating Kubernetes Secrets as encrypted. They are base64, not encrypted. Enable etcd encryption at rest and use an external secrets operator.

Adopting Kubernetes by default. It is the most expensive architecture decision taken with the least analysis.

Exercise 1. The apps/api image weighs 1.3 GB, takes 6 minutes to build in CI, the scanner reports 180 CVEs and the team discovered an npm token visible via docker history. Rewrite the Dockerfile, explaining what each change fixes and what improvement you expect in size, time and findings.

Exercise 2. Reservalia deploys to Kubernetes with kubectl set image from cd.yml, with an administrator kubeconfig stored as a CI secret. Design the migration to GitOps with Argo CD: repository structure, what the pipeline does afterwards, what permissions remain in the CI, how rollback works, and what you lose compared with the current model.

Exercise 3. A four-person startup with a Node monolith and a PostgreSQL database, ~200 users and slow growth, proposes migrating to Kubernetes "to be ready to scale". Write the technical response: what questions to ask first, what the real costs are, what alternatives exist and under what conditions the recommendation would change.

Solution 1.

The indispensable action that is not in the Dockerfile: rotate the npm token. It was in the history of every published image and anybody with registry access could read it. Changing the Dockerfile stops it happening again; it does not repair what happened. It is exactly the lesson from 06-04, and the underlying answer is the same: if the token can be replaced by federated publishing, that is better than rotating it.

A warning about distroless: there is no shell, so the team needs kubectl debug and good observability before adopting it, or the first incident will be unpleasant.

Solution 2.

Repository structure — two, deliberately separated:

They are separated because they have different life cycles and permissions: in deployments you can require platform team reviewers for overlays/production without blocking day-to-day development.

What the pipeline does afterwards:

Production is promoted with a PR from overlays/staging to overlays/production — reviewed and approved — which replaces the Environment approval and leaves the same audit trail, but in Git.

Permissions left in the CI: only a token with write permission on the deployments repository. The administrator kubeconfig disappears from the CI, which is the main gain: compromising the CI no longer amounts to compromising the cluster, only to being able to propose a change that additionally goes through review in production.

Rollback: git revert of the commit in deployments and Argo CD reconciles within the polling interval (or immediately with a webhook). Added benefit: the rollback is recorded as a commit with an author and a reason, rather than as a re-run job nobody remembers. For emergencies you keep kubectl rollout undo as a manual escape hatch, with selfHeal temporarily disabled if necessary — and with the discipline that afterwards you have to fix Git, or the agent will revert your fix.

What you lose: immediacy (seconds to minutes, unless you use a webhook); the simplicity of a single repository, and with it the direct answer to "which commit is in production?", which now requires looking at two histories and is therefore worth automating with an annotation in the manifest; one more component to operate and update; and a learning curve for the team. The maths comes out in favour mainly because of the credentials point, not because of elegance.

Solution 3.

Questions before answering anything:

1. What specific problem do you have today? If the answer is "none, it is just in case", there is no case. If it is "deployments scare us" or "we do not know how to scale during spikes", those problems have cheaper solutions.
2. What is the real load and how variable is it? With 200 users and slow growth, autoscaling adds nothing.
3. Who operates the cluster and what do they stop doing meanwhile? With four people, it is between half and one full-time equivalent.
4. What is the availability target? If there is no written SLO, there is no way to justify complexity on availability grounds.
5. What is there today and what exactly is failing?

Real costs, which are almost never counted:

Alternatives, ordered by what they solve:

Recommendation: do not migrate now. Containerise the application — that yes, because it gives reproducibility and keeps the door open: a containerised application with IaC moves to Kubernetes later in weeks, not months — and deploy on a managed service. Write the decision down with the date and with the conditions that would trigger a review, so it does not get reopened every quarter.

Conditions under which it would change: reaching five or more services with different teams; a contractual on-premise or multi-cloud requirement; a load with spikes of more than an order of magnitude where bin-packing saves measurable money; hiring somebody with real operating experience; or needing ecosystem capabilities (operators, advanced progressive delivery) that the managed service does not offer.

And the argument that usually closes the conversation: "being ready to scale" is the most expensive justification in engineering. With 200 users, the real risk is not being unable to scale — that gets fixed when it happens, and with money — it is spending three months of the team's time on infrastructure while the product stands still. Effective preparation is containerising and having infrastructure as code; the orchestrator gets chosen when the problem exists.

Docker and Kubernetes do not compete with the tools from the previous lessons: they sit underneath them and in front of them. Docker appears in two roles that have to be kept separate — reproducible pipeline environment and immutable artifact that gets deployed — and the separation is realised with multi-stage, minimal runtime images, a non-root user and mounted secrets that leave no trace in the history. BuildKit adds what genuinely speeds up a pipeline on ephemeral runners: mount caches that survive lockfile changes and a layer cache in the registry with mode=max. And the decision about how to build images inside a container — privileged DinD, host socket, or daemonless builders — is a first-order security decision that is usually taken out of inertia.

In Kubernetes, what you have to master in order to deploy is a small set: Deployment, Service, Ingress, ConfigMap, Secret, and above all the probes, because readiness is what makes the rolling update from 03-04 safe and a badly configured liveness probe is what turns an incident into a total outage. kubectl rollout status --timeout is the line that prevents false green, and undo gives the fastest rollback in the course. For parameterising per environment, Helm and Kustomize solve the same thing differently and coexist well. And GitOps won for a specific reason and not because of fashion: it takes the cluster credentials out of the CI and turns the desired state into something explicit, versioned and with drift detection.

The section worth remembering longest is the last one: Kubernetes is the most expensive architecture decision taken with the least analysis. Reservalia is better off on ECS today, and being able to justify that is as valuable as being able to write a Deployment.

The last tool in the tour is the one the student has been using for five modules. GitHub Actions in depth closes the gaps that were left: contexts and expressions, all the triggers still missing and the critical difference between pull_request and pull_request_target, GITHUB_TOKEN permissions and OIDC, the three types of action with the complete code of one written in JavaScript, dynamic matrices, self-hosted runners with autoscaling, the real service limits and how to debug it. After that, 06-07 will put the six tools in the same table and give the criteria for choosing.