Everything we have built so far in this module protects the cluster at runtime. RBAC decides who can do what (08-01). The securityContext limits what a container can do (08-02). Admission policies prevent deploying anything that breaks the rules (08-03). And network policies control what talks to what and what can get out (08-04).

But all of that rests on an assumption we have never questioned: that the images we run are the ones we think they are.

And that assumption is fragile. registry.rutasnorte.example/bookings-api:2.7.1 is a tag, and a tag can be reassigned: the image pulled yesterday may not be today's. The base image it was built on contains hundreds of packages nobody has reviewed. The dependencies were installed from public repositories during the build. And as things stand, nothing stops somebody with registry access from pushing a modified image under the same name and having the cluster run it without complaint, with all our securityContext settings perfectly applied... to a container that does something different from what we think.

This lesson deals with the supply chain of images: where they come from, how they are built, how they are identified unambiguously, how they are signed and how to get the cluster to reject anything that has not been verified.

Important warning. The image policy of a production platform must be designed and reviewed by a security professional. When the images process personal data —bookings-api, bookings-postgres and occupancy-reports at Rutas Norte— the compliance officer must know what controls exist over the software that handles that data, because a compromised dependency is a direct route to it. This lesson's approach is exclusively defensive: understanding where an image can be poisoned in order to close each point and detect the attempts.

1. The supply chain of an image
2. Building minimal images
3. A non-root user in the Dockerfile
4. Identifying the image unambiguously: tags and digests
5. imagePullPolicy and the reused-tag trap
6. The private registry
7. Signing and verification with Cosign
8. Verifying in the cluster with an admission policy
9. SBOM and provenance attestations
10. Base image policy and periodic rebuilds
11. Rutas Norte's complete image policy
12. Common mistakes and tips
13. Exercises
14. Conclusion

A container image does not appear out of nowhere. It is the result of a chain of steps, and every link is a point where something unwanted can be introduced.

Notice that the defences reinforce one another. The digest guarantees that you pull exactly the content you expect, but it does not say whether that content is good. The signature says somebody trusted approved it, but not whether it contains vulnerabilities. The scan says which vulnerabilities it has, but not whether somebody modified it. You need all of them.

Think about bookings-api. It is a Node.js application with perhaps 400 transitive dependencies: packages installed by a package installed by a package you asked for. Every one of them is code that will run with access to the bookings-postgres connection, that is, with access to the customer table.

No kernel bug and no privilege escalation is needed. A compromised dependency is already inside, with the application's legitimate permissions. Not even the most restrictive securityContext prevents it, because it is not doing anything forbidden: it is using the database connection the application needs.

That is the reason this lesson exists.

The principle is simple: what is not in the image cannot have vulnerabilities and cannot be used by anybody.

An image based on a full distribution brings hundreds of packages your application does not use: compilers, package managers, curl, wget, a complete shell, network tools. Each one is attack surface and a potential source of vulnerability alerts you have to manage.

This is the fundamental technique: use an image with tooling to build, and copy only the result into a clean image.

Points worth highlighting:

• npm ci instead of npm install. ci installs exactly what package-lock.json says and fails if the lock file is out of sync. install may resolve different versions on each build, which means two builds of the same code produce different images. Reproducibility and security go hand in hand.
• Copying the manifests before the code. It takes advantage of the layer cache: as long as the dependencies do not change, they are not reinstalled.
• --omit=dev: development dependencies (test frameworks, linters) must not end up in production. They are usually half the tree.
• --chown on the COPY: the files end up with the correct owner without needing a RUN chown that would double the layer's size.
• OCI labels: they leave a trace of where the image came from. When somebody finds this image in the registry a year from now, they will know which repository it came out of.

An important detail that is constantly forgotten: secrets used during the build stay in the layers even if you delete them afterwards. If you do RUN echo $TOKEN > /tmp/token && ... && rm /tmp/token, the token is still in the intermediate layer and anybody who pulls the image can extract it. The correct solution is RUN --mount=type=secret, which mounts the secret during the command without persisting it:

And with nginx, the web-store case:

This is the part the enthusiastic articles do not tell you. With distroless or scratch:

There is no shell. That is exactly what we want from a security standpoint —whoever manages to run code has no tools— but it means the usual diagnostic techniques do not work.

The solution is the one we saw in 07-06: ephemeral containers.

The ephemeral container brings the tools and, with --target, shares the target container's process namespace. You can inspect the process, its file system through /proc/1/root and its network, without the production image containing a single utility.

An important security note: kubectl debug requires the pods/ephemeralcontainers subresource, which in 08-01 we classified as very high risk and did not grant to support or to development. It is a platform operation. And in 08-03 the mandatory-registry Kyverno policy included ephemeralContainers precisely so that nobody could inject an arbitrary image by this route.

A useful principle: scratch for static binaries (Go, Rust), distroless for languages with a runtime (Node.js, Python, Java), alpine when you need a bit of operating system, and a full distribution only if you have a concrete reason you can write down.

In 08-02 we configured runAsNonRoot: true and runAsUser in the manifests. Here we do the same from the other side: declaring the user in the image.

It looks redundant and it is not. They reinforce each other:

The most valuable case is the second: if tomorrow somebody copies this image into a new manifest without the complete securityContext, the Dockerfile's USER still protects. And in the other direction, runAsNonRoot: true in the manifest detects if somebody rebuilt the image without the USER and rejects the pod instead of silently starting it as root.

This is critical and we already flagged it in 08-02:

With a name, Kubernetes cannot verify runAsNonRoot: true, because to resolve it, it would have to read the image's /etc/passwd before starting it:

The pod does not start. A baffling failure whose cause lies in the Dockerfile, not in the manifest.

In distroless there is no adduser. Images with the :nonroot suffix already have user 65532 created, which is the most convenient. For scratch, you copy in an /etc/passwd fabricated in the build stage:

The resulting image contains three files: the binary, the certificates and the /etc/passwd. Nothing else. There is no shell to open, no curl to use, no package manager to install anything. It is the smallest attack surface possible.

A detail that surprises people: without the root certificates, every HTTPS connection fails with a certificate verification error. It is the most common problem when moving to scratch.

With distroless this fails because there is no id. The alternative is to inspect the metadata:

And a check that can be automated in the pipeline:

This is probably the part of the lesson with the most practical consequences.

latest does not mean "the latest one": it is simply the tag used by default when you do not specify one. There is no guarantee whatsoever that it points at anything in particular.

The problems, one by one:

That second point is especially insidious because it produces incidents that are impossible to diagnose: half the requests work and the other half do not, depending on which replica they land on.

In 08-03 we wrote a ValidatingAdmissionPolicy that forbids latest cluster-wide. That policy is what turns this recommendation into a guarantee.

The next step up is to use tags that, by convention and by registry configuration, are never reassigned:

Common schemes:

The key word is immutable: once 2.7.1 is published, that tag is never reassigned to different content. If something has to be fixed, 2.7.2 is published. Many registries let you enforce that rule server-side (section 6), and at that point it stops being a convention and becomes a guarantee.

Even with tags that are immutable by convention, a tag is a mutable pointer: if somebody with registry access reassigns it, the cluster will pull something else.

The digest is the SHA-256 hash of the image manifest. It is cryptographically immutable: if the content changes, the digest changes. It cannot be reassigned because it is not a name, it is a fingerprint of the content.

The practical objection is obvious: a digest is unreadable. The solution is to write both:

Or, in modern practice, let a tool manage it. Kustomize (10-04) resolves the digest automatically:

And tools such as Renovate or Dependabot update the digests with a pull request when new versions appear, so that an image change goes through code review like any other change. That is exactly the goal.

That second command is pure gold when investigating an incident: it tells you exactly what content is running, not which tag was requested. If the digest does not match the one you expected, you have a problem to investigate urgently.

A very useful check: verifying that all the replicas are running the same digest.

Two lines for two containers. If three lines appeared for api, there are replicas with different versions.

imagePullPolicy determines when the kubelet pulls the image from the registry.

If you do not specify imagePullPolicy:

That behaviour has a logic to it: with a digest, IfNotPresent is perfectly safe because the local copy is necessarily the correct content (if the digest matches, the content matches).

Here is the scenario you have to understand:

1. bookings-api:2.7.1 is deployed with imagePullPolicy: IfNotPresent.
2. Nodes A, B and C pull the image and cache it.
3. Somebody reassigns the 2.7.1 tag in the registry to different content.
4. A pod is recreated on node D, which did not have the image: it pulls the new one.
5. The pods on A, B and C carry on running the old one.

The result: replicas of the same Deployment running different content, under the same image name. And no Kubernetes tool will tell you: kubectl get pods shows the same image for all of them.

Only by looking at the imageID (the effective digest) does the discrepancy show up. That is why the check in the previous section is so valuable.

And that is why the underlying solution is using digests, not tweaking imagePullPolicy:

Note the nuance in the first row. Always does not protect you from a maliciously reassigned tag: it guarantees that all the pods run the new content, which is exactly what whoever reassigned it wanted.

That last point is a real security benefit: with IfNotPresent, a node that already has the image cached keeps using it even if the registry credentials have been revoked.

With a digest, IfNotPresent is both safe and robust. It is the best of both worlds.

Exception: in rutas-norte-dev, where you iterate quickly with mutable tags, Always is reasonable. And a warning from 08-02 that applies here: on minikube with locally built images, Always makes the kubelet try to pull from the registry and fail; that is where Never or IfNotPresent make sense.

In 08-03 we wrote the Kyverno policy that requires every image to come from registry.rutasnorte.example. The registry stops being a convenience and becomes the point where all the controls are applied.

We already saw imagePullSecrets in 03-02. The important improvement is declaring them on the ServiceAccount instead of on each pod:

Advantages of declaring it on the ServiceAccount:

• It does not have to be repeated in every Deployment: it gets forgotten less.
• Changing the credentials is a single object.
• You can have one ServiceAccount with access to one repository in the registry and another with access to a different one.

And a warning from 08-01 that applies directly: this Secret contains registry credentials. Anyone with list on secrets in the namespace can read them. In our design, only platform.

The principle is the separation between pushing and pulling:

Production nodes must never hold push credentials. If somebody compromises a node and the imagePullSecret allows pushing, they can upload poisoned images to the registry the whole platform draws from. It is an escalation from one node to the entire cluster.

Most modern registries let you configure a tag so that, once published, it cannot be reassigned:

This turns "we use immutable tags by convention" into a technical guarantee. An attempt to reassign a tag fails:

Using digests is still recommended, because immutability protects against mistakes and against reassignment, but a registry compromised at the storage level could get around it. The digest is verified by the client itself.

Rutas Norte uses postgres:16.4, redis:7.4-alpine and nginx-unprivileged. All of them are excellent public images, and even so it is worth copying them into our own registry:

What this solves:

That last point has an inventory value that gets underestimated: when a serious vulnerability appears in a library, the question will be "which of our images contain it?", and having everything in your own registry turns the answer into a query rather than an investigation.

So far we have guaranteed what content runs (the digest). What is missing is guaranteeing that somebody trusted approved that content.

A digest tells you the image has not changed. It does not tell you whether that image was built by your pipeline or by somebody who stole the registry credentials. The cryptographic signature answers that question: who vouches for this image.

Sigstore is a project that makes software artefact signing accessible. Cosign is its tool for signing container images.

Installation:

The classic model:

Always sign by digest, never by tag. If you sign image:2.7.1, Cosign resolves the digest at that moment; but if somebody reassigns the tag afterwards, the signature does not correspond to the new content and verification fails, which is correct but confusing. Signing by digest makes it clear what is being vouched for.

The problem with a key pair is where the private key lives. If it is in the pipeline, whoever compromises the pipeline can sign anything. It can be mitigated with a KMS:

This is the model Sigstore recommends and it solves the key custody problem.

It works like this:

1. Cosign obtains an OIDC token from an identity provider (the pipeline itself, GitHub Actions, GitLab CI, Google, and so on).
2. It asks Fulcio (Sigstore's certificate authority) for a short-lived certificate, valid for ten minutes, binding the identity to an ephemeral key.
3. It signs the image with that key.
4. It publishes the signature and the certificate in Rekor, a public, immutable, append-only transparency log.
5. It discards the private key.

Verification by identity, not by key:

What you are verifying is not "this image was signed by key X", but something far more useful: "this image was signed by the pipeline of the platform/* repository at git.rutasnorte.example".

Comparison:

Recommendation for Rutas Norte: keyless with OIDC, using the pipeline's identity provider. It is more secure and it removes the work of looking after keys.

A privacy note: Sigstore's public transparency log makes the names of signed images public. If that is a problem (it reveals internal project names), you can deploy a private instance of Fulcio and Rekor.

The order matters: scan before publishing. Publishing first and scanning afterwards means that for a while there is a vulnerable image available for somebody to deploy.

Signing in the pipeline is half the job. If the cluster accepts any image, the signature is decoration: whoever can bypass the pipeline and push directly to the registry deploys whatever they like.

Signature verification has to happen in the cluster, at admission control. It is the only way to guarantee that absolutely everything that runs has been through the process.

We pick up Kyverno from 08-03, now with verifyImages:

Key aspects of the manifest:

• mutateDigest: true is the most elegant part of this policy. Kyverno verifies the signature, obtains the digest and rewrites the manifest so that it uses the digest instead of the tag. It automatically solves the problem from section 4: even though the Deployment says :2.7.1, the pod that is created uses @sha256:..., the very one that was verified.
• required: true: if the image matches no verification rule, it is rejected. Without this, an image from an unforeseen registry would go through unverified.
• useCache: true: without a cache, every pod creation queries the transparency log, which adds latency and creates an external dependency on the critical path.
• webhookTimeoutSeconds: 30: verification can take a while. If the timeout expires and failurePolicy is Fail, pods cannot be created.
• Two rules: our images are verified by OIDC identity; the mirrored public ones, with the key of the platform team that reviewed and copied them.

We asked for a tag and a verified digest is running. Exactly what we wanted.

Sigstore has its own admission controller, specialised in just this:

And it is enabled by labelling the namespace, just like PSA:

For Rutas Norte, Kyverno, because we already have it from 08-03 and it avoids adding another admission webhook. The policy-controller makes sense if you do not want a general policy engine.

This is one of those policies that can halt the platform:

• If Rekor does not respond and there is no cache, verification fails.
• If the pipeline's identity changes (a migration to another CI provider), all new signatures stop validating.
• If failurePolicy: Fail and Kyverno goes down, no pod can be created.

Mandatory mitigations before setting Enforce:

1. Start with validationFailureAction: Audit for weeks.
2. useCache: true always.
3. Kyverno with three replicas, a PDB and anti-affinity (08-03).
4. A documented procedure for disabling the policy in an emergency, with who can do it and how it is logged.
5. Exclude kube-system and rutas-norte-sistema.

An SBOM (Software Bill of Materials) is the complete list of everything inside an image: every system package, every library, every transitive dependency, with its version and its licence.

Standard formats: SPDX (an ISO standard) and CycloneDX (OWASP). Either will do; what matters is having one.

Attaching it to the image as a signed attestation:

That way the SBOM travels with the image, signed, and can be retrieved at any time:

This is the scenario that justifies the whole effort. On an ordinary Tuesday a critical vulnerability is published in a widely used logging library. The question the whole organisation needs to answer in minutes, not days, is:

Is that library in any of our images? In which ones? At which version? Which of them are in production?

Without an SBOM, the answer means pulling every image, inspecting it, reviewing every repository's dependency files and hoping nothing was missed. Days of work, with uncertainty at the end.

With an SBOM:

Two minutes. And with certainty, not with hope.

SLSA (Supply-chain Levels for Software Artifacts) is a framework that defines levels of assurance about how an artefact was built. A provenance attestation answers:

The SLSA levels, in summary:

A reasonable target for a platform like Rutas Norte is L2: an automated build on a hosted service, with signed provenance. L3 requires a build service with strong isolation guarantees, which usually means dedicated infrastructure.

And the piece that closes the circle: requiring the attestation at admission.

With this, an image with no SBOM is not deployed in production. The practical consequence is that the inventory never goes out of date, because it is impossible to deploy something that is not inventoried.

An image built today with node:22.7.0-alpine has no known vulnerabilities. Three months from now, that same image —without anybody having touched it— will have several, because vulnerabilities will have been published in packages it contains.

An image does not degrade: the world changes around it.

This has a consequence many people do not internalise: if an image has been in production for six months without being rebuilt, it accumulates six months of unpatched vulnerabilities, even if the application code is perfect.

The solution is to rebuild regularly, even if the code has not changed.

This is the part of the argument you have to be able to defend when somebody asks "why rebuild if we have not changed anything?".

That last point is the decisive one. A rebuild procedure that runs every week is tested. One that runs once a year, during a crisis, is untested at exactly the moment it matters most.

And tools that automate the tracking:

We bring everything together in the document that governs the platform.

An image is only deployed in production if it meets all of these requirements, and each one is verified by a specific mechanism:

This manifest brings together the whole module: the ServiceAccount from 03-06 and 08-01, the securityContext from 08-02, restricted compliance from 08-03, and the verified digest reference from this lesson.

Using latest in any environment that is not a local experiment. It breaks rollbacks, makes deployments irreproducible and can leave replicas on different versions.

Using moving tags such as :2.7 or :16. They are reassigned with every patch. They look like specific versions and they are not.

Believing that imagePullPolicy: Always protects you from a reassigned tag. It guarantees that every replica runs the new content, which is exactly what whoever reassigned it wanted. The solution is the digest.

Declaring USER by name in the Dockerfile. runAsNonRoot: true cannot verify it and the pod does not start, with an error whose cause lies in another file.

Leaving secrets in the image layers. Deleting them in a later RUN does not remove them from the earlier layer. Use RUN --mount=type=secret.

Using npm install instead of npm ci. It can resolve different versions on each build, making the image irreproducible.

Including development dependencies in the final image. Test frameworks and linters have no business in production and are usually half the dependency tree.

Moving to distroless with no debugging plan. There is no shell. You have to master kubectl debug with ephemeral containers before you need it during an incident.

Forgetting the root certificates when using scratch. Every HTTPS connection fails with a certificate verification error.

Signing by tag instead of by digest. It leaves ambiguity about what content was vouched for.

Storing the private signing key in the pipeline. Whoever compromises the CI can sign anything. Use keyless signing with OIDC, or a KMS.

Signing in the pipeline and not verifying in the cluster. A signature with no verification is decoration: whoever can push directly to the registry deploys whatever they like.

Putting the verification policy on Enforce from day one. It can halt the entire platform. Audit for weeks, useCache: true, high availability for the engine and a documented emergency procedure.

Scanning after publishing. During that interval there is a vulnerable image available to deploy.

Giving push credentials to the production nodes. A compromised node can poison the registry the whole platform draws from.

Not rebuilding the images that do not change. They accumulate months of unpatched vulnerabilities. Rebuild weekly.

Golden tip: the question you must be able to answer at any moment is "what exactly is running in production right now, who approved it and what does it contain?". The digest for the what, the signature for the who, the SBOM for the contents. If one of the three is missing, you have a gap.

This is notifications-worker's current Dockerfile:

1. List all the security and quality problems it has.
2. Rewrite it with a multi-stage build, a minimal base and a non-root user.
3. Write the commands that would verify that the resulting image meets the Rutas Norte policy requirements.

The web-store Deployment in production uses:

1. Explain the three concrete risks of this configuration.
2. Write the commands to obtain the digest, verify its signature and inspect its SBOM.
3. Write the corrected manifest fragment.
4. How would you check that the three replicas are running the same content?

Rutas Norte adds sms-gateway, a component that sends SMS notices about timetable changes. It is built in a different repository (https://git.rutasnorte.example/communications/sms-gateway) by a different team.

Write the Kyverno rule that has to be added to the verify-image-signatures policy so that:

• Only images from registry.rutasnorte.example/rutasnorte/sms-gateway* are accepted.
• Signed from the communications/sms-gateway repository (and not from another one).
• With an attached SBOM containing at least one package.
• Replacing the tag with the verified digest.

Also state how you would roll it out without risking blocking legitimate deployments, and what you would check before moving it to Enforce.

Solution 1

1. Problems with the original Dockerfile:

Problem 9 is subtle and very real: with the shell form, PID 1 is the shell, which does not propagate SIGTERM to the Node process. Kubernetes waits out the full terminationGracePeriodSeconds and then kills the pod with SIGKILL, cutting off the notifications in flight.

2. Rewritten Dockerfile:

And the .dockerignore, which solves problems 4 and 5:

.env and .git on that list are not optional: COPY . . with no .dockerignore can put development credentials and the repository's entire history inside the image, where they stay forever.

3. Verification:

From 412 packages in the original version to 187: less than half the surface to watch.

Solution 2

1. The three risks:

2. Commands:

A note: the date says 2026-07-28, nine days ago. Within the policy's 30-day window, but worth bearing in mind.

3. Corrected manifest:

And with Kustomize (10-04), so as not to write the digest by hand in every environment:

4. Checking that the three replicas run the same thing:

And the version that returns a single value if everything is right:

A 1 means every replica is running the same content. Any higher number is an incident to investigate immediately: it means there are replicas with different contents under the same name.

Solution 3

Decisions worth highlighting:

• A subject with no wildcard. Writing communications/* would allow any repository in that group to sign sms-gateway images. By pinning the exact path, only that specific repository's pipeline can vouch for these images. It is the same least-privilege principle from 08-01, applied to signing.
• A separate rule, not an extension of the existing one. We could have added communications/* to the main rule's subject, but that would let the communications team sign bookings-api images. Each repository signs its own.
• The sms-gateway* pattern with an asterisk covers variants such as sms-gateway-worker if the component grows. If you prefer to be strict, drop the asterisk.

Risk-free rollout:

What to check before Enforce:

Point 7 deserves emphasis. A signature verification policy on Enforce can prevent any deployment if Rekor does not respond or if the pipeline's identity changes. There must be a written procedure, with names, for disabling it within minutes, and that procedure must have been rehearsed. A security measure that can paralyse the platform and has no emergency exit is an operational risk, not a protection.

We have protected the supply chain of Rutas Norte's images:

• An image can be poisoned at five points: the base image, the dependencies, the build process, the registry and the pull. Each one needs its defence, and the defences reinforce one another: the digest guarantees the content, the signature guarantees the endorsement, the SBOM guarantees the inventory.
• Minimal images with multi-stage builds: scratch for static binaries, distroless for languages with a runtime, alpine when you need an operating system. The cost is debugging, which is solved with ephemeral containers (kubectl debug).
• A numeric non-root user in the Dockerfile, reinforced by the manifest's runAsNonRoot (08-02): each one covers the other's gap.
• latest is unacceptable and so are moving tags such as :1.9. An immutable tag is acceptable; the digest is the only cryptographically reproducible reference.
• imagePullPolicy: Always does not protect you from a reassigned tag: it guarantees that every replica runs the new content. With a digest, IfNotPresent is safe and does not depend on the registry responding.
• The private registry is the single control point: imagePullSecrets on the ServiceAccount, a strict separation between pushing and pulling, server-side tag immutability, and mirroring the public images you use.
• Cosign signs the images, preferably keyless with an OIDC identity, which removes the problem of looking after the private key and lets you verify who signed and from where.
• Signing is not enough: you have to verify in the cluster. The Kyverno policy with verifyImages and mutateDigest: true rejects what is unsigned and replaces the tag with the verified digest.
• The SBOM turns "do we have that vulnerable library?" into a two-minute query instead of days of investigation. Provenance attestations (SLSA) answer where the image came from and who built it.
• Rebuilding weekly is cheaper than patching in a hurry: a smaller delta, less risk, and above all a tested procedure rather than one that gets its first outing on the day of the crisis.
• Rutas Norte's complete policy establishes who publishes where and the eleven requirements an image must meet to enter rutas-norte-pro, each one verified by a specific mechanism.

The platform is now protected in all four dimensions: who can do what, what a container can do, what talks to what, and what exactly runs.

But two questions remain unanswered, and they are precisely the ones any audit asks. The first is "who did what?": if tomorrow we discover that the Secret with the bookings-postgres credentials was read, or that a Deployment vanished, or that a ServiceAccount did something odd at three in the morning, right now we have no way of knowing. We have put doors in place, but we keep no record of who goes through them. The second is "what holes do I have right now?": we know how to scan an image before publishing it, but the ones that have been in production for months accumulate new vulnerabilities every week, and nobody is looking at them.

The module's last lesson, 08-06, Auditing, Scanning and Vulnerability Management, answers both: the apiserver audit log and how to query it to answer concrete questions, continuous scanning with Trivy inside and outside the cluster, assessment against the CIS standards with kube-bench, runtime detection with Falco integrated with module 7's Alertmanager, and —what is genuinely missing in most teams— vulnerability management as a process, with deadlines, owners and exceptions that carry an expiry date.