In the previous lesson we hardened every Rutas Norte component: web-store runs without root with a read-only file system, bookings-api has not a single Linux capability, bookings-postgres writes to its volume thanks to fsGroup and all of them carry seccompProfile: RuntimeDefault. The work is done and verified.

And even so, there is an enormous hole: all of that depends on somebody remembering. Tomorrow a colleague adds a sidecar and forgets the drop: ["ALL"]. A developer copies a manifest from a blog that carries privileged: true because "that's how it worked". A third-party operator installs itself with a DaemonSet that mounts the node's /. None of those things breaks anything visible, nobody gets an alert, and the hardening we built so carefully stops being applied at the exact point where it matters.

This lesson takes the leap that changes everything: moving from "each team sets its securityContext properly" to "the cluster rejects a pod that lacks it". It is the third gate we saw in 08-01, admission control, and it is the difference between a written rule and an enforced one.

Important warning. Admission policies are the mechanism that enforces the security design, and for that very reason a mistake in them can block legitimate deployments or, worse, give a false sense of protection. The policy set of a production cluster must be designed and reviewed by a security professional. When the affected workloads handle personal data —bookings-postgres, bookings-api and occupancy-reports at Rutas Norte— the compliance officer must know and approve what is required and which exceptions exist.

Contents

  1. From the written rule to the enforced rule
  2. Historical context: PodSecurityPolicy and why it disappeared
  3. The Pod Security Standards: privileged, baseline and restricted
  4. Pod Security Admission: enabling it with namespace labels
  5. Progressive rollout at Rutas Norte
  6. Workloads that legitimately need privileges
  7. The limitations of Pod Security Admission
  8. General policy engines: Kyverno and OPA Gatekeeper
  9. Kyverno policies for Rutas Norte
  10. ValidatingAdmissionPolicy: native CEL with nothing to install
  11. Common mistakes and tips
  12. Exercises
  13. Conclusion

  1. From the written rule to the enforced rule

Let us recall the journey of an API request from 08-01:

flowchart LR
    A["kubectl apply -f pod.yaml"] --> B["Authentication<br/>who are you?"]
    B --> C["RBAC authorization<br/>may you create pods?"]
    C --> D["Mutating admission<br/>modifies the object"]
    D --> E["Validating admission<br/>does the object comply?<br/>THIS LESSON"]
    E -->|Does not comply| X["Rejected<br/>the object NEVER reaches etcd"]
    E -->|Complies| F[("etcd")]
    style E fill:#d5e8f9,stroke:#36c

The difference from RBAC is fundamental and it is worth being crystal clear about:

RBAC (08-01) Admission control (this lesson)
Question Do you have the right to perform this operation? Is the object you are sending acceptable?
Looks at Subject, verb, resource, namespace The object's content
Example decision "Anna may create pods in dev" "This pod may not be privileged"
If it fails 403 Forbidden 400 Bad Request or 422 with the reason

Somebody from platform has RBAC permission to create pods in rutas-norte-pro. That does not change. What we are adding is that, even with permission, the specific pod they send must meet a set of rules.

And there is a very valuable property: the rejected object never reaches etcd. It is not created and then deleted; it simply does not exist. The answer reaches whoever ran the kubectl apply instantly, with the reason spelled out.

The two kinds of admission controller

Type What it does Examples
Mutating Modifies the object before storing it Injecting the default ServiceAccount, adding a service mesh sidecar, applying the values of a LimitRange (03-05)
Validating Only accepts or rejects, does not modify Pod Security Admission, ResourceQuota (03-04), Kyverno validate policies

The mutating ones run before the validating ones, which makes sense: first the object is completed, then the final result is checked.

  1. Historical context: PodSecurityPolicy and why it disappeared

If you go looking for information on this topic you will find a great deal of material about PodSecurityPolicy (PSP). It matters that you know what they were, because they turn up constantly in old documentation, in forum answers and in inherited manifests. But it also matters that you are absolutely clear about this:

PodSecurityPolicy was deprecated in Kubernetes 1.21 and removed entirely in 1.25. On a 1.30 cluster it does not exist. If a tutorial tells you to create a PodSecurityPolicy, that tutorial is at least five years old.

What they were

A cluster-level resource that described what was allowed in a pod: whether it could be privileged, which users it could use, which volumes, which capabilities. Their shape looked like this:

# DO NOT USE: a resource removed in Kubernetes 1.25. Historical reference only.
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: restrictive
spec:
  privileged: false
  allowPrivilegeEscalation: false
  requiredDropCapabilities: ["ALL"]
  runAsUser:
    rule: MustRunAsNonRoot
  seLinux:
    rule: RunAsAny
  fsGroup:
    rule: MustRunAs
    ranges: [{min: 1, max: 65535}]
  volumes: ["configMap", "emptyDir", "secret", "persistentVolumeClaim"]

Conceptually the idea was good. The implementation had three serious problems that turned out to be insurmountable.

Problem 1: counter-intuitive RBAC

A PSP applied to nobody on its own. It was "enabled" by granting the use verb on it through RBAC, and —this is the odd part— to the subject that created the pod, which very often was not the person but a controller. When a Deployment created pods, the relevant subject was the ReplicaSet controller's ServiceAccount, not the user who ran the kubectl apply.

The result: nobody knew for certain which policy was being applied to a given pod, and answering "why was this rejected?" was an exercise in archaeology.

Problem 2: unpredictable ordering

If several PSPs applied to one subject —which was common— Kubernetes picked one according to a complicated algorithm: first the ones that did not mutate the pod, and if all of them mutated, the first in alphabetical order. Alphabetical order. Renaming a policy changed which policy applied.

Problem 3: mutation

PSPs did not just validate: they also modified the pod (filling in runAsUser, adding fsGroup, dropping capabilities). That meant the pod that ran was not the one in your manifest, and that applying the same YAML on two clusters could give different results with no warning whatsoever.

And an underlying problem: usability

In practice, enabling PSP on an existing cluster broke almost everything, and since there was no "warn me but do not block" mode, the usual route was to create a "temporary" permissive PSP that stayed forever. The community's conclusion was clear: a security mechanism that nobody manages to enable protects nobody.

What was learned

The replacement, Pod Security Admission, was designed explicitly to correct every one of those defects:

PSP problem How PSA solves it
Counter-intuitive RBAC It is enabled with namespace labels. No RBAC involved
Unpredictable ordering Three fixed profiles defined by the project. There is nothing to choose
Mutation It never mutates. It only validates
All or nothing Three modes: warn, audit and enforce. It can be adopted gradually
Complexity One label on a namespace

The trade-off is that PSA is far less flexible: only three profiles, with no way to customise them. That gap is filled by the policy engines in section 8.

  1. The Pod Security Standards: privileged, baseline and restricted

The Pod Security Standards (PSS) are three profiles defined by the Kubernetes project. They are not a resource or an object: they are a specification, an agreement on what "secure" means at three levels.

Profile Philosophy What for
privileged No restrictions System infrastructure: CNI, CSI, node agents
baseline Prevents the known privilege escalations Ordinary applications; realistic, compatible adoption
restricted Current hardening best practice The target for every application

Detailed table of what each profile controls

Control privileged baseline restricted
privileged: true Allowed Forbidden Forbidden
hostNetwork Allowed Forbidden Forbidden
hostPID / hostIPC Allowed Forbidden Forbidden
hostPath Allowed Forbidden Forbidden
hostPort Allowed Forbidden Forbidden
Added capabilities All Only a short list (see below) Only NET_BIND_SERVICE
capabilities.drop: ["ALL"] Not required Not required Mandatory
allowPrivilegeEscalation Free Free Must be false
runAsNonRoot Free Free Must be true
runAsUser: 0 Allowed Allowed Forbidden
seccompProfile Free Cannot be Unconfined RuntimeDefault or Localhost
Volume types All All but the host ones A restricted list
Unsafe sysctls Allowed Forbidden Forbidden
procMount: Unmasked Allowed Forbidden Forbidden
SELinux: dangerous types Allowed Forbidden Forbidden
AppArmor Unconfined Allowed Forbidden Forbidden
/dev/... as a hostPath Allowed Forbidden Forbidden

Capabilities allowed in baseline

baseline allows adding only these, which are the ones in the default set and all relatively harmless:

AUDIT_WRITE, CHOWN, DAC_OVERRIDE, FOWNER, FSETID, KILL,
MKNOD, NET_BIND_SERVICE, SETFCAP, SETGID, SETPCAP, SETUID, SYS_CHROOT

What baseline forbids adding is the genuinely dangerous set: NET_ADMIN, SYS_ADMIN, SYS_MODULE, SYS_PTRACE, BPF, NET_RAW...

restricted goes much further: it requires drop: ["ALL"] and only allows adding back NET_BIND_SERVICE. That is exactly the exception we discussed in 08-02, and it is why we said the high port is preferable: with port 8080 you do not even need that one.

Volume types allowed in restricted

configMap, csi, downwardAPI, emptyDir, ephemeral, persistentVolumeClaim,
projected, secret

Notice that every type Rutas Norte uses is there —emptyDir for the writable directories, persistentVolumeClaim for the PostgreSQL data, configMap and secret for configuration— and hostPath is not. It is the formalisation of the warning from 05-01.

The minimum securityContext that satisfies restricted

spec:
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: app
      image: registry.rutasnorte.example/bookings-api:2.7.1
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]

Compare it with the "golden tip" block from 08-02: it is practically the same. The Pod Security Standards are nothing other than the official codification of what we already did by hand. Everything from the previous lesson now fits into a label.

An important note: restricted does not require readOnlyRootFilesystem. It is a known omission (it was too disruptive for many workloads) and one of the reasons a policy engine complements PSA.

  1. Pod Security Admission: enabling it with namespace labels

The Pod Security Admission (PSA) is the controller that applies the Pod Security Standards. It has been built into the apiserver since 1.25: there is nothing to install, no pod to maintain, no webhook that can go down.

It is enabled by putting labels on the Namespace:

apiVersion: v1
kind: Namespace
metadata:
  name: rutas-norte-pro
  labels:
    app.kubernetes.io/part-of: rutas-norte
    environment: pro
    # --- Pod Security Admission ---
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: v1.30
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: v1.30
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: v1.30

The label syntax is:

pod-security.kubernetes.io/<MODE>[-version]: <VALUE>

The three modes

Mode What happens when a pod breaches Where it shows up
enforce The pod is rejected. It is not created An immediate error on apply
audit It is created, but recorded in the apiserver's audit log The audit log (08-06)
warn It is created, and the client receives a warning kubectl output

The three are independent and can be combined. That is the key to gradual adoption: you can set enforce: baseline (blocks the worst) together with warn: restricted and audit: restricted (which tell you what still does not comply, without blocking).

An essential detail about enforce:

enforce only acts on pods that are created or updated. It does not affect those already running. If you enable enforce: restricted in a namespace with non-compliant pods, those pods carry on happily... until the Deployment creates a new one, which will fail. It is a deferred and baffling failure if you are not expecting it.

That is why warn and audit do not block but do evaluate the objects that create pods (Deployments, StatefulSets, CronJobs), warning you at the moment of the apply. enforce, by contrast, only evaluates the pod, not the Deployment. We will come back to this in section 7.

Pinning the version

pod-security.kubernetes.io/enforce-version: v1.30

The Pod Security Standards evolve: each Kubernetes release may add new controls. If you do not pin the version, the default is latest, which means a cluster upgrade can start rejecting pods that used to be accepted, without anyone having touched a thing.

Value Behaviour
latest (the default) Always uses the controls of the cluster's current version
v1.30 Freezes the controls of that version

Always pin the version. And when you upgrade the cluster, raise the label as an explicit, reviewed change, with warn first. It is the difference between a planned upgrade and a Monday-morning incident.

Setting cluster-wide defaults

You can tell the apiserver a default profile for namespaces that carry no labels, through AdmissionConfiguration:

# /etc/kubernetes/admission/pod-security.yaml (on the control plane nodes)
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
  - name: PodSecurity
    configuration:
      apiVersion: pod-security.admission.config.k8s.io/v1
      kind: PodSecurityConfiguration
      defaults:
        enforce: "baseline"          # baseline for unlabelled namespaces
        enforce-version: "v1.30"
        audit: "restricted"
        audit-version: "v1.30"
        warn: "restricted"
        warn-version: "v1.30"
      exemptions:
        usernames: []
        runtimeClasses: []
        namespaces:
          - kube-system              # the system needs privileges

This is very powerful because it covers future namespaces: if tomorrow somebody creates rutas-norte-experiments with no labels, it starts out on baseline already. Without this configuration, an unlabelled namespace is equivalent to privileged, that is, with no restriction at all.

It requires access to the control plane, so on managed Kubernetes (10-06) it may not be available; in that case, the alternative is a Kyverno policy that requires the labels on every new namespace (we will see it in section 9).

The warn mode in action

kubectl label namespace rutas-norte-pre \
  pod-security.kubernetes.io/warn=restricted \
  pod-security.kubernetes.io/warn-version=v1.30

kubectl apply -f k8s/environments/pre/web-store.yaml
Warning: would violate PodSecurity "restricted:v1.30": allowPrivilegeEscalation != false
(container "nginx" must set securityContext.allowPrivilegeEscalation=false),
unrestricted capabilities (container "nginx" must set securityContext.capabilities.drop=["ALL"]),
runAsNonRoot != true (pod or container "nginx" must set securityContext.runAsNonRoot=true),
seccompProfile (pod or container "nginx" must set securityContext.seccompProfile.type
to "RuntimeDefault" or "Localhost")
deployment.apps/web-store configured

Read that output carefully: the Deployment has been applied (configured), but the warning lists exactly the four missing fields, container by container. It is the best possible documentation of what needs fixing. Note how well written the message is: each line tells you the field, the container and the expected value.

  1. Progressive rollout at Rutas Norte

Never switch on enforce: restricted in one go in a production namespace. The correct path has four phases and can take weeks. Let us walk it.

flowchart TD
    A["Phase 0<br/>No labels<br/>= privileged"] --> B["Phase 1<br/>warn + audit: restricted<br/>DOES NOT block"]
    B --> C["Analyse the warnings<br/>and fix components"]
    C --> D["Phase 2<br/>enforce: baseline<br/>+ warn/audit: restricted"]
    D --> E["Fix what is missing<br/>for restricted"]
    E --> F["Phase 3<br/>enforce: restricted<br/>in all three modes"]
    style F fill:#d5f9d5,stroke:#3a3

Phase 1: see what would break, without breaking anything

kubectl label namespace rutas-norte-pro \
  pod-security.kubernetes.io/warn=restricted \
  pod-security.kubernetes.io/warn-version=v1.30 \
  pod-security.kubernetes.io/audit=restricted \
  pod-security.kubernetes.io/audit-version=v1.30

Nothing is blocked here: the cluster carries on exactly as before. Now the existing workloads must be forced through an evaluation. A very useful trick is to reapply the manifests unchanged:

kubectl apply -f k8s/environments/pro/ --dry-run=server 2>&1 | grep -i warning

--dry-run=server sends the object to the apiserver, which passes it through the whole admission chain and returns the result without storing it. It is the safe way to assess the full picture.

Warning: would violate PodSecurity "restricted:v1.30": allowPrivilegeEscalation != false
  (container "postgres" ...), runAsNonRoot != true (pod or container "postgres" ...)
Warning: would violate PodSecurity "restricted:v1.30": host namespaces (hostPath volume "varlog"),
  restricted volume types (volume "varlog" uses restricted volume type "hostPath")

And to see what the audit mode evaluates, the source is the apiserver's audit log, with the pod-security.kubernetes.io/audit-violations annotation. We will look at it in detail in 08-06.

A tool that saves a lot of time in this phase:

# Evaluates the whole cluster against a profile without applying anything
kubectl krew install score   # or use pod-security-admission-checker

You can also query directly which namespaces are unprotected:

kubectl get namespaces -o custom-columns=\
'NAME:.metadata.name,ENFORCE:.metadata.labels.pod-security\.kubernetes\.io/enforce,WARN:.metadata.labels.pod-security\.kubernetes\.io/warn'
NAME                ENFORCE       WARN
default             <none>        <none>
kube-system         privileged    <none>
rutas-norte-dev     <none>        <none>
rutas-norte-pre     <none>        restricted
rutas-norte-pro     <none>        restricted

The <none> values in enforce are namespaces where anything at all can be deployed right now, default included. That table makes an excellent slide for a security meeting.

Phase 2: enforce: baseline

baseline blocks the genuinely dangerous things —privileged pods, host namespaces, hostPath— and hardly any ordinary application breaches it. It is a step with very little risk and a lot of value.

# k8s/environments/pro/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: rutas-norte-pro
  labels:
    app.kubernetes.io/part-of: rutas-norte
    environment: pro
    pod-security.kubernetes.io/enforce: baseline     # blocks the worst
    pod-security.kubernetes.io/enforce-version: v1.30
    pod-security.kubernetes.io/audit: restricted     # still reports against the target
    pod-security.kubernetes.io/audit-version: v1.30
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: v1.30

Now, if somebody tries to deploy a privileged pod:

kubectl run test --image=nginx --privileged -n rutas-norte-pro
Error from server (Forbidden): pods "test" is forbidden: violates PodSecurity
"baseline:v1.30": privileged (container "test" must not set securityContext.privileged=true)

That is the moment when the work of 08-02 stops depending on anybody's memory.

Phase 3: enforce: restricted and the exact rejection message

Before taking this step, every component must comply. Let us first try an unhardened manifest to see the full message:

# /tmp/unhardened-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: restricted-test
  namespace: rutas-norte-pro
spec:
  containers:
    - name: app
      image: registry.rutasnorte.example/utilities:1.4.2
kubectl label namespace rutas-norte-pro \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/enforce-version=v1.30 --overwrite

kubectl apply -f /tmp/unhardened-pod.yaml
Error from server (Forbidden): error when creating "/tmp/unhardened-pod.yaml":
pods "restricted-test" is forbidden: violates PodSecurity "restricted:v1.30":
allowPrivilegeEscalation != false (container "app" must set
securityContext.allowPrivilegeEscalation=false),
unrestricted capabilities (container "app" must set securityContext.capabilities.drop=["ALL"]),
runAsNonRoot != true (pod or container "app" must set securityContext.runAsNonRoot=true),
seccompProfile (pod or container "app" must set securityContext.seccompProfile.type
to "RuntimeDefault" or "Localhost")

The message is a to-do list. We fix it:

# /tmp/hardened-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: restricted-test
  namespace: rutas-norte-pro
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 10001
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: app
      image: registry.rutasnorte.example/utilities:1.4.2
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]
kubectl apply -f /tmp/hardened-pod.yaml
pod/restricted-test created

The component that did not comply: bookings-postgres

In 08-02 we left readOnlyRootFilesystem: false on PostgreSQL as a documented exception. Good news: restricted does not require that field, so that point does not block. What it does require is runAsNonRoot: true and drop: ["ALL"], which we already had. bookings-postgres passes restricted unchanged.

Let us check before touching the namespace:

kubectl apply -f k8s/environments/pro/bookings-postgres.yaml --dry-run=server
statefulset.apps/bookings-postgres configured

No warnings. Perfect.

The component that does not comply: the log collector

kubectl apply -f k8s/environments/pro/logs-daemonset.yaml --dry-run=server
Warning: would violate PodSecurity "restricted:v1.30": hostPath volumes (volume "varlog"),
runAsNonRoot != true (pod or container "collector" must set securityContext.runAsNonRoot=true),
restricted volume types (volume "varlog" uses restricted volume type "hostPath")

And here there is nothing to fix: the collector legitimately needs to read the node's /var/log and needs to be root in order to do so. That is the subject of the next section.

Final state of the three environments

Namespace enforce audit / warn Reason
rutas-norte-dev baseline restricted Developers need room to debug
rutas-norte-pre restricted restricted It must be identical to production
rutas-norte-pro restricted restricted The target
rutas-norte-sistema privileged Node agents (section 6)

That rutas-norte-pre is just as strict as production is non-negotiable: if pre-production allows what production does not, the promotion fails at the worst possible moment. That is the entire point of having an intermediate environment.

rutas-norte-dev on baseline is a deliberate concession: it blocks the dangerous things but allows, for example, running as root to try something out. With warn: restricted, the developer sees on every apply exactly what will be missing for promotion. It is a balance decision that has to be documented.

  1. Workloads that legitimately need privileges

There is software that cannot comply with restricted, and not out of laziness. The CNI plugin configures the node's network interfaces. The CSI driver mounts volumes on the host file system. The log collector reads /var/log. The runtime detection agent (Falco, which we will see in 08-06) observes system calls.

The correct answer is not to relax the profile of rutas-norte-pro. It is to isolate those workloads.

Solution 1: a separate namespace with the privileged profile

apiVersion: v1
kind: Namespace
metadata:
  name: rutas-norte-sistema
  labels:
    app.kubernetes.io/part-of: rutas-norte
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/enforce-version: v1.30
    # audit and warn on restricted: this leaves a trace of everything that skips the profile
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: v1.30
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: v1.30
  annotations:
    security.rutasnorte.example/justification: >-
      Namespace for node agents that require host access:
      log collector (reading /var/log) and runtime detection
      agent. Quarterly review by the platform team
      and by security. No business application may be deployed here.

Three important decisions in that manifest:

  1. audit and warn stay on restricted. Even though enforce is privileged, every pod that skips the profile leaves a trace in the audit log. That way the privileged namespace is not a black hole: we know exactly what runs there and with what privileges.
  2. The justification annotation. A namespace with the privileged profile is a security exception and must be documented on the object itself, with who reviews it and when.
  3. Strict RBAC on top. Remembering 08-01: whoever can create pods here can create privileged pods. Only platform should have permission, and in 08-01 we saw that creating pods in a namespace is equivalent to being able to use any ServiceAccount in that namespace: another reason for the applications not to live there.

And the essential RBAC counterpart:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: platform-system
  namespace: rutas-norte-sistema
subjects:
  - kind: Group
    name: platform           # ONLY platform. Not development, not CI, not support
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: admin
  apiGroup: rbac.authorization.k8s.io

Solution 2: exemptions in the apiserver configuration

PSA accepts global exemptions on three criteria:

apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
  - name: PodSecurity
    configuration:
      apiVersion: pod-security.admission.config.k8s.io/v1
      kind: PodSecurityConfiguration
      defaults:
        enforce: "baseline"
        enforce-version: "v1.30"
      exemptions:
        # By user: the subject that creates the pod is exempt
        usernames:
          - "system:serviceaccount:kube-system:daemon-set-controller"
        # By runtime class: pods with this RuntimeClass are exempt
        runtimeClasses:
          - "gvisor"
        # By namespace: nothing in the namespace is evaluated
        namespaces:
          - "kube-system"
          - "rutas-norte-sistema"
Criterion When to use it Risk
namespaces Whole infrastructure namespaces Medium: you must watch who can deploy there
usernames One specific controller that creates system pods High: that identity is exempt across the whole cluster
runtimeClasses Pods that already run sandboxed (gVisor, Kata) Low: the isolation comes from the runtime

A warning about usernames: it exempts the subject in every namespace, not just where you want it. It is a very broad permission. The per-namespace exemption is easier to reason about and to audit.

And the general warning: every exemption is a security exception. It must have a written justification, an owner, and a periodic review confirming that it is still needed. At Rutas Norte the exemption list is reviewed every quarter along with RBAC.

  1. The limitations of Pod Security Admission

PSA is excellent at what it does, and what it does is quite narrow. Knowing its limits saves you from trusting it for things it does not do.

Limitation 1: it only looks at the pod

PSA evaluates Pod objects. Full stop. It does not look at Services, or Ingresses, or PVCs, or ConfigMaps, or NetworkPolicies.

This has an uncomfortable practical effect with enforce:

# A Deployment with a pod that does NOT comply with restricted
kubectl apply -f bad-deployment.yaml
deployment.apps/bad-application created
kubectl get pods -n rutas-norte-pro -l app=bad-application
No resources found in rutas-norte-pro namespace.
kubectl describe replicaset -n rutas-norte-pro -l app=bad-application | tail -5
Events:
  Type     Reason        Age   From                   Message
  ----     ------        ----  ----                   -------
  Warning  FailedCreate  12s   replicaset-controller  Error creating: pods
  "bad-application-6c5f9b8d4-" is forbidden: violates PodSecurity "restricted:v1.30":
  allowPrivilegeEscalation != false (container "app" must set
  securityContext.allowPrivilegeEscalation=false), ...

The Deployment is created successfully, the ReplicaSet is created, and it is the ReplicaSet that fails to create pods. Whoever ran the apply sees a success and has to go and look at the events to discover the problem.

That is why warn is so valuable: it does evaluate the objects that create pods and warns at the moment of the apply. Always keep warn enabled even when you have enforce, precisely for this reason.

Limitation 2: it cannot require anything outside the securityContext

Things PSA cannot require:

It cannot require Why it matters at Rutas Norte
Labels (app.kubernetes.io/part-of) Without them the selectors and the module 7 dashboards break
resources.requests and limits Without them the QoS is BestEffort (03-05) and the pod is the first to be evicted
A specific image registry Anybody can deploy from Docker Hub without review
That the image does not use latest Irreproducible deployments (08-05)
Health probes (liveness, readiness) Without them nothing from module 7 works
readOnlyRootFilesystem Not even restricted requires it
That a NetworkPolicy exists The microsegmentation of 08-04
automountServiceAccountToken: false What we worked on in 03-06

That list is exactly the gap Kyverno and Gatekeeper fill.

Limitation 3: the three profiles are not configurable

You cannot create a profile that is "restricted but allowing read-only hostPath" or "restricted plus readOnlyRootFilesystem". The profiles are defined by the Kubernetes project and they are immutable.

Limitation 4: it does not mutate

It is a design virtue (it avoids PSP's problem 3), but it means PSA will never help you by filling in fields. If you want every pod to get seccompProfile: RuntimeDefault automatically, you need a mutating policy engine.

Summary

Need Tool
Prevent privileged pods, hostPath, root PSA (built in, nothing to install)
Require labels, limits, an image registry Kyverno or Gatekeeper
Fill in fields automatically Kyverno (mutation)
Verify image signatures Kyverno or the Sigstore policy-controller (08-05)
Simple rules with nothing to install ValidatingAdmissionPolicy (section 10)

The practical recommendation: PSA always, as a baseline that cannot fail (it lives in the apiserver and depends on no pod), and a policy engine on top for everything else.

  1. General policy engines: Kyverno and OPA Gatekeeper

A policy engine is an admission webhook: the apiserver sends it every object and it answers whether it accepts, rejects or modifies it.

flowchart LR
    A["kubectl apply"] --> B["apiserver"]
    B --> C["PSA<br/>(built in)"]
    C --> D["Mutating webhook<br/>Kyverno"]
    D --> E["Validating webhook<br/>Kyverno / Gatekeeper"]
    E -->|Rejects| X["Error with the reason"]
    E -->|Accepts| F[("etcd")]

This brings a very serious operational consideration: the policy engine becomes a dependency of the apiserver. If the webhook is configured with failurePolicy: Fail and the engine's pods are down, nobody can create anything in the cluster. Whole clusters have gone down that way.

failurePolicy If the webhook does not respond When to use it
Ignore The object is accepted Initial adoption, non-critical policies
Fail The object is rejected Security policies, with a highly available engine

The usual mitigations: several engine replicas with anti-affinity (06-05), a PodDisruptionBudget (09-05), and excluding kube-system from the webhooks so that the cluster can be recovered.

Comparison: Kyverno and OPA Gatekeeper

Kyverno OPA Gatekeeper
Policy language YAML (like Kubernetes) Rego (OPA's own language)
Learning curve Low: if you know YAML, you can write policies High: Rego is a different declarative language
Validate Yes Yes
Mutate Yes, very well resolved Yes, more limited
Generate objects Yes (create a NetworkPolicy with each namespace) No
Verify image signatures Yes, integrated with Cosign Requires additional work
Resource cleanup Yes (CleanupPolicy) No
Compliance reports PolicyReport as a CRD Violation objects
Scope Kubernetes only Generic: also CI, Terraform, APIs
Ready-made policies A broad, maintained catalogue A template library
Community and maturity CNCF, fast growth CNCF, more established
Performance with many policies Good Very good

Recommendation for Rutas Norte: Kyverno. The reasons are concrete:

  • The team already writes YAML all day. Learning Rego would be a real barrier to developers writing policies too.
  • Signature verification with Cosign is built in, and that is exactly what we will need in 08-05.
  • The ability to generate objects allows, for example, automatically creating the deny-all NetworkPolicy from 04-06 in every new namespace.
  • Mutation solves the "add seccompProfile to everything" problem without touching a hundred manifests.

Gatekeeper is the better option if the organisation already uses OPA for other things (API access control, Terraform policies) and wants a single policy language for everything.

Installing Kyverno

helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update

helm install kyverno kyverno/kyverno \
  --namespace kyverno --create-namespace \
  --set admissionController.replicas=3 \
  --set admissionController.podDisruptionBudget.minAvailable=2

Three replicas and a PDB: the policy engine is critical infrastructure and must be treated as such.

  1. Kyverno policies for Rutas Norte

A Kyverno policy is a ClusterPolicy (the whole cluster) or a Policy (one namespace) with a list of rules.

Policy 1: every image from the company registry

This is probably the highest-value policy per line written. It prevents anything being deployed that has not gone through the company registry, where in 08-05 and 08-06 we will put the scanning and the signing.

# k8s/policies/mandatory-registry.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: mandatory-registry
  annotations:
    policies.kyverno.io/title: Mandatory image registry
    policies.kyverno.io/category: Supply chain security
    policies.kyverno.io/severity: high
    policies.kyverno.io/description: >-
      Every image deployed in the Rutas Norte namespaces must come from
      registry.rutasnorte.example. Public images are mirrored there
      beforehand, where they are scanned and signed. See lessons 08-05 and 08-06.
spec:
  # Fail: if Kyverno cannot evaluate, it rejects. This is a security policy.
  validationFailureAction: Enforce
  background: true          # also evaluates existing objects, for the reports
  rules:
    - name: check-registry
      match:
        any:
          - resources:
              kinds:
                - Pod
              namespaces:
                - rutas-norte-dev
                - rutas-norte-pre
                - rutas-norte-pro
      validate:
        message: >-
          The image "{{ request.object.spec.containers[0].image }}" does not come from
          registry.rutasnorte.example. Every image must be mirrored into the
          company registry, where images are scanned and signed.
          See the platform image guide.
        pattern:
          spec:
            # The = sign means "if the field exists, it must comply"
            =(ephemeralContainers):
              - image: "registry.rutasnorte.example/*"
            =(initContainers):
              - image: "registry.rutasnorte.example/*"
            containers:
              - image: "registry.rutasnorte.example/*"

Let us take the manifest apart:

  • validationFailureAction: Enforce rejects. The alternative is Audit, which only produces a report. Just as with PSA, you start on Audit and move to Enforce when there are no breaches left.
  • background: true makes Kyverno also evaluate objects that already exist, producing PolicyReport objects. Without it you would only see new objects.
  • match.any.resources defines the scope. Here, pods in the three Rutas Norte namespaces. kyverno, kube-system and rutas-norte-sistema are left out because they use external infrastructure images.
  • =(initContainers) and =(ephemeralContainers): the =() prefix means "if this field exists, it must match the pattern". Without it, a pod with no initContainers would fail validation. Do not forget the ephemeral containers: they are the kubectl debug route (07-06) and without this line somebody could inject an arbitrary image into a production pod.
  • The message appears verbatim in the terminal of whoever is deploying. Writing it well —what fails, why, and what to do— saves an enormous number of questions.

Trying it out:

kubectl run test --image=nginx:latest -n rutas-norte-pro
Error from server: admission webhook "validate.kyverno.svc-fail" denied the request:

resource Pod/rutas-norte-pro/test was blocked due to the following policies

mandatory-registry:
  check-registry: 'validation error: The image "nginx:latest" does not come from
    registry.rutasnorte.example. Every image must be mirrored into the company
    registry, where images are scanned and signed. See the platform image
    guide. rule check-registry failed at path /spec/containers/0/image/'

Policy 2: require the labels of the official scheme

In 02-07 we defined the Rutas Norte label scheme, and in module 7 the dashboards and the alerts depend on it. A workload without app.kubernetes.io/part-of: rutas-norte is invisible to monitoring.

# k8s/policies/mandatory-labels.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: mandatory-labels
  annotations:
    policies.kyverno.io/title: Rutas Norte label scheme
    policies.kyverno.io/category: Governance
    policies.kyverno.io/severity: medium
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: labels-on-workloads
      match:
        any:
          - resources:
              kinds:
                - Deployment
                - StatefulSet
                - DaemonSet
                - CronJob
                - Job
              namespaces:
                - rutas-norte-dev
                - rutas-norte-pre
                - rutas-norte-pro
      validate:
        message: >-
          Mandatory labels are missing. Every Rutas Norte workload must carry,
          in metadata and in the pod template: "app" (the component name),
          "app.kubernetes.io/part-of: rutas-norte" and "environment" (dev|pre|pro).
          Without them, the workload appears in neither the dashboards nor the alerts.
        pattern:
          metadata:
            labels:
              app: "?*"                                  # ?* = not empty
              app.kubernetes.io/part-of: "rutas-norte"   # exact value
              environment: "dev | pre | pro"             # one of the three
          spec:
            template:
              metadata:
                labels:
                  app: "?*"
                  app.kubernetes.io/part-of: "rutas-norte"
                  environment: "dev | pre | pro"

    - name: environment-namespace-consistency
      match:
        any:
          - resources:
              kinds: [Deployment, StatefulSet, DaemonSet, CronJob]
              namespaces: [rutas-norte-pro]
      validate:
        message: >-
          In rutas-norte-pro the "environment" label must be "pro".
          An incorrect label makes the production alerts get
          routed to the wrong channel (see 07-04).
        pattern:
          metadata:
            labels:
              environment: "pro"

The Kyverno pattern syntax that appears here:

Syntax Meaning
"?*" Any non-empty value
"rutas-norte" That exact value
"dev | pre | pro" One of those three (the | is an "or")
"registry.rutasnorte.example/*" A trailing wildcard
=(field) If the field exists, it must comply
X(field) The field must not exist

The second rule is a detail that looks minor and is not: an environment: dev label on a manifest copied into production makes Alertmanager route the alerts to the development channel and nobody hears about a real outage. The policy prevents it.

Policy 3: require resource limits

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: mandatory-resources
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: requests-and-limits
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaces: [rutas-norte-pre, rutas-norte-pro]
      validate:
        message: >-
          Every container must declare CPU and memory requests and limits.
          Without them the QoS class is BestEffort (see 03-05) and the pod is the
          first to be evicted when the node is short of resources.
        pattern:
          spec:
            containers:
              - resources:
                  requests:
                    cpu: "?*"
                    memory: "?*"
                  limits:
                    memory: "?*"

Notice that we require limits.memory but not limits.cpu. It is deliberate and it is a debated decision: limiting the CPU causes throttling that can degrade bookings-api latency without it being obvious why, whereas not limiting memory can take the node down. It is a good example of a policy that codifies a technical decision made by the team; we will cover it thoroughly in 09-06.

Policy 4: mutate to add the security baseline

This is where Kyverno shines compared with PSA:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-security-baseline
spec:
  rules:
    - name: default-seccomp
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaces: [rutas-norte-dev, rutas-norte-pre, rutas-norte-pro]
      mutate:
        patchStrategicMerge:
          spec:
            # +() means "add this value ONLY if the field does not exist"
            +(securityContext):
              seccompProfile:
                type: RuntimeDefault

With +(), if the manifest already defines securityContext, Kyverno leaves it alone; if it does not, it adds it. It is a safety net, not a replacement for the work of 08-02: the right thing is still to declare it explicitly in the manifest, so that whoever reads it knows what is running.

A warning about mutating: it is exactly problem 3 of PodSecurityPolicy. Use it sparingly and only for safe defaults, never to "fix" incorrect manifests, because then the repository's YAML stops describing what runs.

Policy 5: generate the deny-all NetworkPolicy in every new namespace

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: generate-deny-all
spec:
  rules:
    - name: deny-all-in-rutas-norte-namespaces
      match:
        any:
          - resources:
              kinds: [Namespace]
              selector:
                matchLabels:
                  app.kubernetes.io/part-of: rutas-norte
      generate:
        apiVersion: networking.k8s.io/v1
        kind: NetworkPolicy
        name: deny-all
        namespace: "{{request.object.metadata.name}}"
        synchronize: true      # if somebody deletes it, Kyverno recreates it
        data:
          spec:
            podSelector: {}
            policyTypes: [Ingress, Egress]

This guarantees that no Rutas Norte namespace can exist without the default deny we built in 04-06. And synchronize: true means that, if somebody deletes it, it comes back on its own. It is a capability Gatekeeper does not have and that solves a real problem.

Consulting the reports

kubectl get policyreport -n rutas-norte-pro
NAME                                   PASS   FAIL   WARN   ERROR   SKIP   AGE
polr-ns-rutas-norte-pro                 47     2      0      0       0      6d
kubectl get policyreport -n rutas-norte-pro -o json | jq -r '
  .items[].results[] | select(.result == "fail")
  | "\(.policy)/\(.rule): \(.resources[0].kind)/\(.resources[0].name)\n    \(.message)"'
mandatory-labels/labels-on-workloads: Deployment/legacy-exporter
    validation error: Mandatory labels are missing...
mandatory-resources/requests-and-limits: Pod/temp-debug-x8k2p
    validation error: Every container must declare requests and limits...

Those reports are the outstanding to-do list and, with background: true, they keep themselves up to date. In 08-06 we will turn them into evidence for compliance audits.

  1. ValidatingAdmissionPolicy: native CEL with nothing to install

Since Kubernetes 1.30, the ValidatingAdmissionPolicy lets you write admission rules inside the apiserver, with no webhook, using CEL (Common Expression Language) expressions.

Advantages over an external engine:

ValidatingAdmissionPolicy Webhook (Kyverno/Gatekeeper)
Installation None Helm, pods, certificates
Availability That of the apiserver It can go down and block the cluster
Latency No network hop One HTTP call per object
Mutation No (there is an alpha MutatingAdmissionPolicy) Yes
Generate objects No Kyverno does
Verify signatures No Kyverno does
Expressiveness CEL: good for field rules Very high

Example: forbid the latest tag

It is made up of two objects: the policy (what is checked) and the binding (where it applies).

# k8s/policies/vap-forbid-latest.yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: forbid-latest-tag
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  validations:
    - expression: >-
        object.spec.containers.all(c,
          !c.image.endsWith(":latest") && c.image.contains(":"))
      message: >-
        Images may not use the "latest" tag and must carry an explicit
        tag. Use an immutable tag or, better still, a digest
        (image@sha256:...). See lesson 08-05.
      reason: Invalid
    - expression: >-
        !has(object.spec.initContainers) ||
        object.spec.initContainers.all(c,
          !c.image.endsWith(":latest") && c.image.contains(":"))
      message: 'The same rule applies to the initContainers.'
      reason: Invalid
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: forbid-latest-tag
spec:
  policyName: forbid-latest-tag
  validationActions: [Deny]       # Deny | Warn | Audit, combinable
  matchResources:
    namespaceSelector:
      matchLabels:
        app.kubernetes.io/part-of: rutas-norte

Points to note:

  • validationActions plays the same role as the PSA modes: [Warn, Audit] to try it out, [Deny] when you are sure. They can be combined: [Deny, Audit].
  • namespaceSelector applies the policy only to the namespaces labelled as part of Rutas Norte. The system ones are left out.
  • CEL has very readable functions: all(), exists(), has(), endsWith(), contains(), matches() for regular expressions.
  • Separating policy and binding lets you write the rule once and apply it with different severity in different environments: [Warn] in dev and [Deny] in pro, with two bindings.

A test:

kubectl run test --image=registry.rutasnorte.example/utilities:latest -n rutas-norte-pro
The pods "test" is invalid: ValidatingAdmissionPolicy 'forbid-latest-tag'
with binding 'forbid-latest-tag' denied request: Images may not use
the "latest" tag and must carry an explicit tag. Use an immutable tag
or, better still, a digest (image@sha256:...). See lesson 08-05.

An example with parameters

VAPs can read configuration from an external resource, which makes them reusable:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: limit-replicas
spec:
  failurePolicy: Fail
  paramKind:
    apiVersion: v1
    kind: ConfigMap
  matchConstraints:
    resourceRules:
      - apiGroups: ["apps"]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["deployments"]
  validations:
    - expression: >-
        object.spec.replicas <= int(params.data.maxReplicas)
      messageExpression: >-
        "The number of replicas (" + string(object.spec.replicas) +
        ") exceeds the maximum allowed (" + params.data.maxReplicas + ")."

messageExpression builds the message with the real values, something you appreciate a great deal when debugging.

Criteria for use

Use VAP for simple rules on fields: forbidding latest, requiring a label, capping a numeric value, checking a prefix. They are free in availability terms and add no piece to maintain.

Use Kyverno for what VAP cannot do: verifying image signatures (08-05), generating objects, mutating, or policies that need to look at other cluster resources.

And always use PSA as the securityContext baseline, because it is built in and cannot fail.

At Rutas Norte we use all three, in layers:

flowchart TD
    A["Every pod that is created"] --> B["PSA: restricted<br/>securityContext baseline<br/>(built in, infallible)"]
    B --> C["VAP: simple rules<br/>no latest, mandatory fields<br/>(built in)"]
    C --> D["Kyverno: registry, signatures,<br/>labels, resources, generation<br/>(webhook)"]
    D --> E["Pod admitted"]
    style B fill:#d5f9d5,stroke:#3a3
    style C fill:#d5e8f9,stroke:#36c
    style D fill:#f9f0d5,stroke:#ca3

Common Mistakes and Tips

Looking for PodSecurityPolicy tutorials. They were removed in 1.25. If a resource talks to you about kind: PodSecurityPolicy, it is out of date and probably everything else in it is too.

Enabling enforce: restricted straight away in production. The existing pods carry on running, but the first restart or the first update fails. Always walk the phases: warn/audit, then baseline, then restricted.

Not pinning -version in the labels. Without it latest is used and a cluster upgrade can reject pods that used to pass. Pin the version and raise it as an explicit change.

Believing that enforce warns when you apply a Deployment. It does not: PSA only evaluates pods. The Deployment is created and it is the ReplicaSet that fails, in an event you have to go looking for. Always keep warn enabled, since it does evaluate the objects that create pods.

Leaving namespaces unlabelled. A namespace with no PSA labels is equivalent to privileged. Check default and any namespace created by hand; configure the default in the apiserver or require the labels with Kyverno.

Relaxing a whole namespace's profile because of one workload. If the log collector needs hostPath, it goes to rutas-norte-sistema with the privileged profile; you do not drop rutas-norte-pro to baseline.

Forgetting the RBAC of the privileged namespace. A namespace with the privileged profile where development can create pods is worse than having no PSA: it gives a false sense of security. And remember from 08-01 that creating pods there allows using any ServiceAccount in that namespace.

Using failurePolicy: Fail without a highly available engine. If Kyverno goes down, nobody can create anything. Three replicas, a PDB and anti-affinity as a minimum, and exclude kube-system.

Forgetting initContainers and ephemeralContainers in the policies. An initContainer with an unverified image bypasses the entire registry policy. Ephemeral containers are the kubectl debug route.

Setting validationFailureAction: Enforce from day one. Just as with PSA: Audit first, look at the PolicyReport objects, fix, and only then Enforce.

Overusing mutation. If Kyverno "fixes" the manifests, the repository's YAML stops describing what runs. Use it only for safe defaults.

Useless error messages. The message is read by a person who is trying to deploy. Tell them what fails, why the rule exists and what they have to do. A good message saves the platform team an interruption.

Golden tip: policies are code. They live in k8s/policies/ in the repository, they are reviewed in a pull request, and they are tested in rutas-norte-dev before reaching production. A policy applied by hand in production is exactly the same problem as a RoleBinding applied by hand.

Exercises

Exercise 1: prepare a namespace for restricted

The team creates rutas-norte-analytics to deploy an occupancy statistics aggregation service. Write:

  1. The Namespace manifest in phase 1 (warn without blocking) with the version pinned.
  2. The command that would evaluate an existing manifest against the profile without deploying it.
  3. The Namespace manifest in its final state, given that the service is already hardened.

Exercise 2: interpret and fix a rejection

Deploying a new component in rutas-norte-pro, which has enforce: restricted, you get:

Error from server (Forbidden): error when creating "exporter.yaml":
pods "timetable-exporter" is forbidden: violates PodSecurity "restricted:v1.30":
non-default capabilities (container "exporter" must not include "NET_RAW" in
securityContext.capabilities.add), restricted volume types (volume "node-config"
uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container
"exporter" must set securityContext.runAsNonRoot=true), seccompProfile
(pod or container "exporter" must set securityContext.seccompProfile.type
to "RuntimeDefault" or "Localhost")

The original manifest:

apiVersion: v1
kind: Pod
metadata:
  name: timetable-exporter
  namespace: rutas-norte-pro
spec:
  containers:
    - name: exporter
      image: registry.rutasnorte.example/timetable-exporter:2.1.0
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]
          add: ["NET_RAW"]
      volumeMounts:
        - name: node-config
          mountPath: /etc/timetables
  volumes:
    - name: node-config
      hostPath:
        path: /etc/rutasnorte/timetables
        type: Directory
  1. List the four violations and explain what each one means.
  2. Rewrite the manifest so that it complies with restricted, knowing that NET_RAW was added "just in case" and that the file in /etc/rutasnorte/timetables is in fact a static configuration file.
  3. What would you have done if NET_RAW really had been indispensable?

Exercise 3: a Kyverno policy for automountServiceAccountToken

In 03-06 we established that workloads which do not talk to the API must carry automountServiceAccountToken: false. Write a Kyverno ClusterPolicy that:

  • Applies to Pods in rutas-norte-pre and rutas-norte-pro.
  • Rejects pods that do not declare automountServiceAccountToken explicitly (neither true nor false), forcing it to be a conscious decision.
  • Exempts pods whose ServiceAccount is bookings-api or inventory-agent, which do need the token.
  • Includes a message explaining what to do.

Also state how you would roll it out safely.

Solutions

Solution 1

1. Phase 1: warn without blocking

# k8s/environments/analytics/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: rutas-norte-analytics
  labels:
    app.kubernetes.io/part-of: rutas-norte
    environment: pro
    # Phase 1: no enforce. We are only observing.
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: v1.30
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: v1.30
  annotations:
    security.rutasnorte.example/psa-phase: >-
      Phase 1 (warn+audit) since 2026-08-06. Target: enforce:restricted
      before 2026-09-01. Owner: the platform team.

The annotation with a target date stops "temporary" from becoming "permanent", which is how almost every phase 1 ends up.

2. Evaluate without deploying

kubectl apply -f k8s/environments/analytics/aggregator.yaml --dry-run=server
Warning: would violate PodSecurity "restricted:v1.30": allowPrivilegeEscalation != false
(container "aggregator" must set securityContext.allowPrivilegeEscalation=false),
unrestricted capabilities (container "aggregator" must set
securityContext.capabilities.drop=["ALL"])
deployment.apps/aggregator created (server dry run)

--dry-run=server passes the object through the whole admission chain —PSA, VAP and Kyverno included— and returns the verdict without storing anything. It is different from --dry-run=client, which only validates the YAML locally and would detect none of this.

3. Final state

apiVersion: v1
kind: Namespace
metadata:
  name: rutas-norte-analytics
  labels:
    app.kubernetes.io/part-of: rutas-norte
    environment: pro
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: v1.30
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: v1.30
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: v1.30

audit and warn are kept even though enforce already blocks: warn still evaluates the Deployments (which enforce does not see) and audit leaves a record in the audit log.

Solution 2

1. The four violations:

Violation Meaning
non-default capabilities: NET_RAW restricted only allows adding NET_BIND_SERVICE back. NET_RAW (raw sockets) is forbidden
restricted volume types: hostPath hostPath is not on the list of volumes restricted allows. It is the formalisation of the warning from 05-01
runAsNonRoot != true It was not declared; restricted requires the explicit declaration, it is not enough for the image to have a USER
seccompProfile RuntimeDefault is missing. Remember from 08-02 that without declaring it there is probably no filter at all

2. Corrected manifest:

# k8s/environments/pro/timetable-exporter.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: timetables-config
  namespace: rutas-norte-pro
  labels:
    app: timetable-exporter
    app.kubernetes.io/part-of: rutas-norte
    environment: pro
data:
  timetables.yaml: |
    routes:
      - code: N-101
        origin: Bilbao
        destination: Santander
        departures: ["07:00", "10:30", "15:00", "19:45"]
---
apiVersion: v1
kind: Pod
metadata:
  name: timetable-exporter
  namespace: rutas-norte-pro
  labels:
    app: timetable-exporter
    app.kubernetes.io/part-of: rutas-norte
    environment: pro
spec:
  automountServiceAccountToken: false
  securityContext:
    runAsNonRoot: true                # violation 3
    runAsUser: 10005
    runAsGroup: 10005
    seccompProfile:
      type: RuntimeDefault            # violation 4
  containers:
    - name: exporter
      image: registry.rutasnorte.example/timetable-exporter:2.1.0
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true  # restricted does not require it, but it is right
        capabilities:
          drop: ["ALL"]               # violation 1: without the NET_RAW add
      volumeMounts:
        - name: timetables            # violation 2: a ConfigMap instead of hostPath
          mountPath: /etc/timetables
          readOnly: true
        - name: temp
          mountPath: /tmp
      resources:
        requests: { cpu: 50m, memory: 64Mi }
        limits:   { memory: 128Mi }
  volumes:
    - name: timetables
      configMap:
        name: timetables-config
    - name: temp
      emptyDir: { sizeLimit: 32Mi }

The four changes and their justification:

  1. NET_RAW removed. It was "just in case": it allows creating raw sockets, which is useful for ping and for inspecting traffic. A timetable exporter does not need it. If the process fails to start, the log will say so and then we will investigate why; capabilities are not granted pre-emptively.
  2. hostPath replaced with a ConfigMap. The file was static configuration, which is exactly what a ConfigMap is for (03-01). Besides satisfying the policy, this is a gain: the file no longer depends on somebody having copied it onto every node, it is versioned in Git and it is identical in all three environments. Fixing the policy breach improved the design, which is what usually happens.
  3. runAsNonRoot: true + runAsUser: 10005 declared explicitly.
  4. seccompProfile: RuntimeDefault on the pod, inherited by the container.

Extras that restricted did not require but the Kyverno policies from section 9 do: the scheme labels, resources and automountServiceAccountToken: false. Plus readOnlyRootFilesystem: true, which no policy requires but which is the baseline from 08-02.

3. If NET_RAW really were indispensable:

The process would be this, in order:

  1. Question the requirement. Why does a timetable exporter need raw sockets? In 90% of cases there is an alternative (a TCP check instead of ICMP, for instance).
  2. If it is real, use a library that does not need it or change the design.
  3. If there is no alternative, do not relax rutas-norte-pro. It gets deployed in its own namespace with enforce: baseline (which does allow NET_RAW), with restricted RBAC, with an annotated justification, with a quarterly review and with approval from the security professional.
  4. Document the exception in the security exception register, with an expiry date. We will come back to this register in 08-06.

What you never do is drop rutas-norte-pro to baseline: that turns one component's exception into an exception for the whole platform.

Solution 3

# k8s/policies/explicit-serviceaccount-token.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: explicit-serviceaccount-token
  annotations:
    policies.kyverno.io/title: Explicit decision on the ServiceAccount token
    policies.kyverno.io/category: Security
    policies.kyverno.io/severity: medium
    policies.kyverno.io/description: >-
      Every pod must declare automountServiceAccountToken explicitly.
      Mounting the token without needing it hands an API credential to a
      process that does not use it (see 03-06). The workloads that do talk to the API
      are exempted by ServiceAccount name.
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: require-explicit-declaration
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaces:
                - rutas-norte-pre
                - rutas-norte-pro
      exclude:
        any:
          # Workloads that legitimately need the token
          - resources:
              subjects:
                - kind: ServiceAccount
                  name: bookings-api
                - kind: ServiceAccount
                  name: inventory-agent
      preconditions:
        all:
          # Do not apply to the pods the system itself creates
          - key: "{{ request.object.metadata.namespace }}"
            operator: AnyIn
            value: ["rutas-norte-pre", "rutas-norte-pro"]
      validate:
        message: >-
          The pod must declare spec.automountServiceAccountToken
          explicitly. If the workload does NOT talk to the Kubernetes API (the usual case),
          set "automountServiceAccountToken: false". If it does, set it to true
          and make sure its ServiceAccount has a Role with least
          privilege (see 08-01 and 03-06).
        pattern:
          spec:
            automountServiceAccountToken: "false | true"

Notes on the solution:

  • The "false | true" pattern forces the field to exist with one of those two values. If the field is absent, validation fails, which is exactly what the exercise asked for: turning an omission into a decision.
  • exclude.any.resources.subjects allows exempting by ServiceAccount. A more maintainable alternative in the long run would be to exempt by a label (security.rutasnorte.example/uses-api: "true"), so that the policy does not have to be edited every time a new workload needs the API.
  • It is a governance policy, not an attack-blocking one: its value lies in forcing somebody to think on every deployment.

Safe rollout, in four steps:

# 1. Apply in Audit mode: it blocks nothing
sed 's/validationFailureAction: Enforce/validationFailureAction: Audit/' \
  k8s/policies/explicit-serviceaccount-token.yaml | kubectl apply -f -

# 2. Wait for the background scan to complete the report and review it
kubectl get policyreport -A -o json | jq -r '
  .items[].results[]
  | select(.policy == "explicit-serviceaccount-token" and .result == "fail")
  | "\(.resources[0].namespace)/\(.resources[0].name)"'
rutas-norte-pro/notifications-worker-6d4f8b9c7-k2m4x
rutas-norte-pro/occupancy-reports-29187360-9wzqt
rutas-norte-pre/web-store-7c8d9f6b5-p3n8v
# 3. Fix the manifests of those workloads (add the explicit field)
#    and check that the report comes back clean.

# 4. Only then, move to Enforce
kubectl apply -f k8s/policies/explicit-serviceaccount-token.yaml

And the final check:

kubectl run test --image=registry.rutasnorte.example/utilities:1.4.2 -n rutas-norte-pro
Error from server: admission webhook "validate.kyverno.svc-fail" denied the request:

resource Pod/rutas-norte-pro/test was blocked due to the following policies

explicit-serviceaccount-token:
  require-explicit-declaration: 'validation error: The pod must declare
    spec.automountServiceAccountToken explicitly...'

One additional and important precaution: before moving to Enforce, check that the policy does not block the system controllers that create pods in those namespaces (the ReplicaSet controller, the Job controller). If a controller creates pods without the field, your deployments will stop working. Testing it first in rutas-norte-dev with Enforce for a few days is the cheap way to find out.

Conclusion

We have turned the hardening of 08-02 into something the cluster enforces on its own:

  • Admission control is the third gate: RBAC decides whether you have the right to the operation, admission decides whether the object is acceptable. The rejected object never reaches etcd.
  • PodSecurityPolicy is history: deprecated in 1.21, removed in 1.25. Its three defects —counter-intuitive RBAC, unpredictable alphabetical ordering and mutation— explain the design of its replacement.
  • The Pod Security Standards define three profiles: privileged (no restrictions), baseline (blocks the known escalations) and restricted (the target for every application), which is essentially the official codification of what we did by hand in 08-02.
  • Pod Security Admission is built into the apiserver and is enabled with namespace labels, with three modes —enforce, audit, warn— that allow gradual adoption. Always pin -version so that a cluster upgrade breaks nothing.
  • The correct adoption is phased: warn+audit, then enforce: baseline, then enforce: restricted. Never in one go in production. And rutas-norte-pre must be as strict as rutas-norte-pro.
  • Workloads that legitimately need privileges —CNI, CSI, the log collector— go into a separate namespace with the privileged profile, restricted RBAC and an annotated justification. You never relax the production profile because of one workload.
  • PSA has clear limits: it only looks at the pod, it cannot require labels, resource limits or image registries, and it does not mutate.
  • That gap is filled by Kyverno (YAML, mutates, generates, verifies signatures: Rutas Norte's choice) and OPA Gatekeeper (Rego, generic beyond Kubernetes), and the ValidatingAdmissionPolicy objects with CEL cover the simple rules with nothing to install.
  • Rutas Norte uses all three in layers: PSA as the infallible base, VAP for simple rules, Kyverno for the mandatory registry, the scheme labels, the resource limits and the automatic generation of the deny-all NetworkPolicy.

The cluster now rejects, on its own, a privileged pod, one that mounts hostPath, one that runs as root or one whose image does not come from the company registry. Nobody can slip past it by oversight.

But let us look at the whole picture again. We have protected who can do what (08-01) and what a container can do (08-02 and 08-03). An entire dimension is missing: what can talk to what. In 04-06 we put a deny-all NetworkPolicy in rutas-norte-pro and authorised the conversations one at a time, and even then we pointed out two uncomfortable limits: the policies work at L3/L4 —they know nothing about HTTP paths or methods— and they log nothing, so a denied connection attempt is invisible. What is more, inside the cluster all the traffic travels unencrypted once past the Ingress TLS: whoever can observe the node's network sees the queries to bookings-postgres in the clear, personal data included. And outbound traffic is still practically unrestricted, which is the natural route by which a customer database gets exfiltrated.

The next lesson, 08-04, Network Security, builds the complete strategy on top of what you already know: microsegmentation, egress traffic control and the problem of IPs versus domain names, encryption in transit with mTLS and service meshes —Istio, Linkerd and Cilium compared, with an honest criterion for when you need one and when you do not—, protecting the perimeter and the control plane, and the traffic visibility that is exactly what NetworkPolicy lacks.

Kubernetes Course

Module 1: Introduction to Kubernetes

Module 2: Core Kubernetes Components

Module 3: Configuration and Secret Management

Module 4: Networking in Kubernetes

Module 5: Storage in Kubernetes

Module 6: Advanced Kubernetes Concepts

Module 7: Monitoring and Logging

Module 8: Kubernetes Security

Module 9: Scaling and Performance

Module 10: Kubernetes Ecosystem and Tooling

Module 11: Case Studies and Real-World Applications

Module 12: Preparing for Kubernetes Certification

© Copyright 2026. All rights reserved