In the previous lesson we hardened every Rutas Norte component: web-store runs without root with a read-only file system, bookings-api has not a single Linux capability, bookings-postgres writes to its volume thanks to fsGroup and all of them carry seccompProfile: RuntimeDefault. The work is done and verified.
And even so, there is an enormous hole: all of that depends on somebody remembering. Tomorrow a colleague adds a sidecar and forgets the drop: ["ALL"]. A developer copies a manifest from a blog that carries privileged: true because "that's how it worked". A third-party operator installs itself with a DaemonSet that mounts the node's /. None of those things breaks anything visible, nobody gets an alert, and the hardening we built so carefully stops being applied at the exact point where it matters.
This lesson takes the leap that changes everything: moving from "each team sets its securityContext properly" to "the cluster rejects a pod that lacks it". It is the third gate we saw in 08-01, admission control, and it is the difference between a written rule and an enforced one.
Important warning. Admission policies are the mechanism that enforces the security design, and for that very reason a mistake in them can block legitimate deployments or, worse, give a false sense of protection. The policy set of a production cluster must be designed and reviewed by a security professional. When the affected workloads handle personal data —
bookings-postgres,bookings-apiandoccupancy-reportsat Rutas Norte— the compliance officer must know and approve what is required and which exceptions exist.
Contents
- From the written rule to the enforced rule
- Historical context: PodSecurityPolicy and why it disappeared
- The Pod Security Standards:
privileged,baselineandrestricted - Pod Security Admission: enabling it with namespace labels
- Progressive rollout at Rutas Norte
- Workloads that legitimately need privileges
- The limitations of Pod Security Admission
- General policy engines: Kyverno and OPA Gatekeeper
- Kyverno policies for Rutas Norte
- ValidatingAdmissionPolicy: native CEL with nothing to install
- Common mistakes and tips
- Exercises
- Conclusion
- From the written rule to the enforced rule
Let us recall the journey of an API request from 08-01:
flowchart LR
A["kubectl apply -f pod.yaml"] --> B["Authentication<br/>who are you?"]
B --> C["RBAC authorization<br/>may you create pods?"]
C --> D["Mutating admission<br/>modifies the object"]
D --> E["Validating admission<br/>does the object comply?<br/>THIS LESSON"]
E -->|Does not comply| X["Rejected<br/>the object NEVER reaches etcd"]
E -->|Complies| F[("etcd")]
style E fill:#d5e8f9,stroke:#36c
The difference from RBAC is fundamental and it is worth being crystal clear about:
| RBAC (08-01) | Admission control (this lesson) | |
|---|---|---|
| Question | Do you have the right to perform this operation? | Is the object you are sending acceptable? |
| Looks at | Subject, verb, resource, namespace | The object's content |
| Example decision | "Anna may create pods in dev" | "This pod may not be privileged" |
| If it fails | 403 Forbidden |
400 Bad Request or 422 with the reason |
Somebody from platform has RBAC permission to create pods in rutas-norte-pro. That does not change. What we are adding is that, even with permission, the specific pod they send must meet a set of rules.
And there is a very valuable property: the rejected object never reaches etcd. It is not created and then deleted; it simply does not exist. The answer reaches whoever ran the kubectl apply instantly, with the reason spelled out.
The two kinds of admission controller
| Type | What it does | Examples |
|---|---|---|
| Mutating | Modifies the object before storing it | Injecting the default ServiceAccount, adding a service mesh sidecar, applying the values of a LimitRange (03-05) |
| Validating | Only accepts or rejects, does not modify | Pod Security Admission, ResourceQuota (03-04), Kyverno validate policies |
The mutating ones run before the validating ones, which makes sense: first the object is completed, then the final result is checked.
- Historical context: PodSecurityPolicy and why it disappeared
If you go looking for information on this topic you will find a great deal of material about PodSecurityPolicy (PSP). It matters that you know what they were, because they turn up constantly in old documentation, in forum answers and in inherited manifests. But it also matters that you are absolutely clear about this:
PodSecurityPolicy was deprecated in Kubernetes 1.21 and removed entirely in 1.25. On a 1.30 cluster it does not exist. If a tutorial tells you to create a
PodSecurityPolicy, that tutorial is at least five years old.
What they were
A cluster-level resource that described what was allowed in a pod: whether it could be privileged, which users it could use, which volumes, which capabilities. Their shape looked like this:
# DO NOT USE: a resource removed in Kubernetes 1.25. Historical reference only.
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: restrictive
spec:
privileged: false
allowPrivilegeEscalation: false
requiredDropCapabilities: ["ALL"]
runAsUser:
rule: MustRunAsNonRoot
seLinux:
rule: RunAsAny
fsGroup:
rule: MustRunAs
ranges: [{min: 1, max: 65535}]
volumes: ["configMap", "emptyDir", "secret", "persistentVolumeClaim"]Conceptually the idea was good. The implementation had three serious problems that turned out to be insurmountable.
Problem 1: counter-intuitive RBAC
A PSP applied to nobody on its own. It was "enabled" by granting the use verb on it through RBAC, and —this is the odd part— to the subject that created the pod, which very often was not the person but a controller. When a Deployment created pods, the relevant subject was the ReplicaSet controller's ServiceAccount, not the user who ran the kubectl apply.
The result: nobody knew for certain which policy was being applied to a given pod, and answering "why was this rejected?" was an exercise in archaeology.
Problem 2: unpredictable ordering
If several PSPs applied to one subject —which was common— Kubernetes picked one according to a complicated algorithm: first the ones that did not mutate the pod, and if all of them mutated, the first in alphabetical order. Alphabetical order. Renaming a policy changed which policy applied.
Problem 3: mutation
PSPs did not just validate: they also modified the pod (filling in runAsUser, adding fsGroup, dropping capabilities). That meant the pod that ran was not the one in your manifest, and that applying the same YAML on two clusters could give different results with no warning whatsoever.
And an underlying problem: usability
In practice, enabling PSP on an existing cluster broke almost everything, and since there was no "warn me but do not block" mode, the usual route was to create a "temporary" permissive PSP that stayed forever. The community's conclusion was clear: a security mechanism that nobody manages to enable protects nobody.
What was learned
The replacement, Pod Security Admission, was designed explicitly to correct every one of those defects:
| PSP problem | How PSA solves it |
|---|---|
| Counter-intuitive RBAC | It is enabled with namespace labels. No RBAC involved |
| Unpredictable ordering | Three fixed profiles defined by the project. There is nothing to choose |
| Mutation | It never mutates. It only validates |
| All or nothing | Three modes: warn, audit and enforce. It can be adopted gradually |
| Complexity | One label on a namespace |
The trade-off is that PSA is far less flexible: only three profiles, with no way to customise them. That gap is filled by the policy engines in section 8.
- The Pod Security Standards:
privileged, baseline and restricted
privileged, baseline and restrictedThe Pod Security Standards (PSS) are three profiles defined by the Kubernetes project. They are not a resource or an object: they are a specification, an agreement on what "secure" means at three levels.
| Profile | Philosophy | What for |
|---|---|---|
privileged |
No restrictions | System infrastructure: CNI, CSI, node agents |
baseline |
Prevents the known privilege escalations | Ordinary applications; realistic, compatible adoption |
restricted |
Current hardening best practice | The target for every application |
Detailed table of what each profile controls
| Control | privileged |
baseline |
restricted |
|---|---|---|---|
privileged: true |
Allowed | Forbidden | Forbidden |
hostNetwork |
Allowed | Forbidden | Forbidden |
hostPID / hostIPC |
Allowed | Forbidden | Forbidden |
hostPath |
Allowed | Forbidden | Forbidden |
hostPort |
Allowed | Forbidden | Forbidden |
| Added capabilities | All | Only a short list (see below) | Only NET_BIND_SERVICE |
capabilities.drop: ["ALL"] |
Not required | Not required | Mandatory |
allowPrivilegeEscalation |
Free | Free | Must be false |
runAsNonRoot |
Free | Free | Must be true |
runAsUser: 0 |
Allowed | Allowed | Forbidden |
seccompProfile |
Free | Cannot be Unconfined |
RuntimeDefault or Localhost |
| Volume types | All | All but the host ones | A restricted list |
Unsafe sysctls |
Allowed | Forbidden | Forbidden |
procMount: Unmasked |
Allowed | Forbidden | Forbidden |
| SELinux: dangerous types | Allowed | Forbidden | Forbidden |
AppArmor Unconfined |
Allowed | Forbidden | Forbidden |
/dev/... as a hostPath |
Allowed | Forbidden | Forbidden |
Capabilities allowed in baseline
baseline allows adding only these, which are the ones in the default set and all relatively harmless:
AUDIT_WRITE, CHOWN, DAC_OVERRIDE, FOWNER, FSETID, KILL,
MKNOD, NET_BIND_SERVICE, SETFCAP, SETGID, SETPCAP, SETUID, SYS_CHROOTWhat baseline forbids adding is the genuinely dangerous set: NET_ADMIN, SYS_ADMIN, SYS_MODULE, SYS_PTRACE, BPF, NET_RAW...
restricted goes much further: it requires drop: ["ALL"] and only allows adding back NET_BIND_SERVICE. That is exactly the exception we discussed in 08-02, and it is why we said the high port is preferable: with port 8080 you do not even need that one.
Volume types allowed in restricted
Notice that every type Rutas Norte uses is there —emptyDir for the writable directories, persistentVolumeClaim for the PostgreSQL data, configMap and secret for configuration— and hostPath is not. It is the formalisation of the warning from 05-01.
The minimum securityContext that satisfies restricted
spec:
securityContext:
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
containers:
- name: app
image: registry.rutasnorte.example/bookings-api:2.7.1
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]Compare it with the "golden tip" block from 08-02: it is practically the same. The Pod Security Standards are nothing other than the official codification of what we already did by hand. Everything from the previous lesson now fits into a label.
An important note: restricted does not require readOnlyRootFilesystem. It is a known omission (it was too disruptive for many workloads) and one of the reasons a policy engine complements PSA.
- Pod Security Admission: enabling it with namespace labels
The Pod Security Admission (PSA) is the controller that applies the Pod Security Standards. It has been built into the apiserver since 1.25: there is nothing to install, no pod to maintain, no webhook that can go down.
It is enabled by putting labels on the Namespace:
apiVersion: v1
kind: Namespace
metadata:
name: rutas-norte-pro
labels:
app.kubernetes.io/part-of: rutas-norte
environment: pro
# --- Pod Security Admission ---
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/enforce-version: v1.30
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/audit-version: v1.30
pod-security.kubernetes.io/warn: restricted
pod-security.kubernetes.io/warn-version: v1.30The label syntax is:
The three modes
| Mode | What happens when a pod breaches | Where it shows up |
|---|---|---|
enforce |
The pod is rejected. It is not created | An immediate error on apply |
audit |
It is created, but recorded in the apiserver's audit log | The audit log (08-06) |
warn |
It is created, and the client receives a warning | kubectl output |
The three are independent and can be combined. That is the key to gradual adoption: you can set enforce: baseline (blocks the worst) together with warn: restricted and audit: restricted (which tell you what still does not comply, without blocking).
An essential detail about enforce:
enforceonly acts on pods that are created or updated. It does not affect those already running. If you enableenforce: restrictedin a namespace with non-compliant pods, those pods carry on happily... until the Deployment creates a new one, which will fail. It is a deferred and baffling failure if you are not expecting it.
That is why warn and audit do not block but do evaluate the objects that create pods (Deployments, StatefulSets, CronJobs), warning you at the moment of the apply. enforce, by contrast, only evaluates the pod, not the Deployment. We will come back to this in section 7.
Pinning the version
The Pod Security Standards evolve: each Kubernetes release may add new controls. If you do not pin the version, the default is latest, which means a cluster upgrade can start rejecting pods that used to be accepted, without anyone having touched a thing.
| Value | Behaviour |
|---|---|
latest (the default) |
Always uses the controls of the cluster's current version |
v1.30 |
Freezes the controls of that version |
Always pin the version. And when you upgrade the cluster, raise the label as an explicit, reviewed change, with warn first. It is the difference between a planned upgrade and a Monday-morning incident.
Setting cluster-wide defaults
You can tell the apiserver a default profile for namespaces that carry no labels, through AdmissionConfiguration:
# /etc/kubernetes/admission/pod-security.yaml (on the control plane nodes)
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: PodSecurity
configuration:
apiVersion: pod-security.admission.config.k8s.io/v1
kind: PodSecurityConfiguration
defaults:
enforce: "baseline" # baseline for unlabelled namespaces
enforce-version: "v1.30"
audit: "restricted"
audit-version: "v1.30"
warn: "restricted"
warn-version: "v1.30"
exemptions:
usernames: []
runtimeClasses: []
namespaces:
- kube-system # the system needs privilegesThis is very powerful because it covers future namespaces: if tomorrow somebody creates rutas-norte-experiments with no labels, it starts out on baseline already. Without this configuration, an unlabelled namespace is equivalent to privileged, that is, with no restriction at all.
It requires access to the control plane, so on managed Kubernetes (10-06) it may not be available; in that case, the alternative is a Kyverno policy that requires the labels on every new namespace (we will see it in section 9).
The warn mode in action
kubectl label namespace rutas-norte-pre \
pod-security.kubernetes.io/warn=restricted \
pod-security.kubernetes.io/warn-version=v1.30
kubectl apply -f k8s/environments/pre/web-store.yamlWarning: would violate PodSecurity "restricted:v1.30": allowPrivilegeEscalation != false
(container "nginx" must set securityContext.allowPrivilegeEscalation=false),
unrestricted capabilities (container "nginx" must set securityContext.capabilities.drop=["ALL"]),
runAsNonRoot != true (pod or container "nginx" must set securityContext.runAsNonRoot=true),
seccompProfile (pod or container "nginx" must set securityContext.seccompProfile.type
to "RuntimeDefault" or "Localhost")
deployment.apps/web-store configuredRead that output carefully: the Deployment has been applied (configured), but the warning lists exactly the four missing fields, container by container. It is the best possible documentation of what needs fixing. Note how well written the message is: each line tells you the field, the container and the expected value.
- Progressive rollout at Rutas Norte
Never switch on enforce: restricted in one go in a production namespace. The correct path has four phases and can take weeks. Let us walk it.
flowchart TD
A["Phase 0<br/>No labels<br/>= privileged"] --> B["Phase 1<br/>warn + audit: restricted<br/>DOES NOT block"]
B --> C["Analyse the warnings<br/>and fix components"]
C --> D["Phase 2<br/>enforce: baseline<br/>+ warn/audit: restricted"]
D --> E["Fix what is missing<br/>for restricted"]
E --> F["Phase 3<br/>enforce: restricted<br/>in all three modes"]
style F fill:#d5f9d5,stroke:#3a3
Phase 1: see what would break, without breaking anything
kubectl label namespace rutas-norte-pro \
pod-security.kubernetes.io/warn=restricted \
pod-security.kubernetes.io/warn-version=v1.30 \
pod-security.kubernetes.io/audit=restricted \
pod-security.kubernetes.io/audit-version=v1.30Nothing is blocked here: the cluster carries on exactly as before. Now the existing workloads must be forced through an evaluation. A very useful trick is to reapply the manifests unchanged:
--dry-run=server sends the object to the apiserver, which passes it through the whole admission chain and returns the result without storing it. It is the safe way to assess the full picture.
Warning: would violate PodSecurity "restricted:v1.30": allowPrivilegeEscalation != false
(container "postgres" ...), runAsNonRoot != true (pod or container "postgres" ...)
Warning: would violate PodSecurity "restricted:v1.30": host namespaces (hostPath volume "varlog"),
restricted volume types (volume "varlog" uses restricted volume type "hostPath")And to see what the audit mode evaluates, the source is the apiserver's audit log, with the pod-security.kubernetes.io/audit-violations annotation. We will look at it in detail in 08-06.
A tool that saves a lot of time in this phase:
# Evaluates the whole cluster against a profile without applying anything
kubectl krew install score # or use pod-security-admission-checkerYou can also query directly which namespaces are unprotected:
kubectl get namespaces -o custom-columns=\
'NAME:.metadata.name,ENFORCE:.metadata.labels.pod-security\.kubernetes\.io/enforce,WARN:.metadata.labels.pod-security\.kubernetes\.io/warn'NAME ENFORCE WARN
default <none> <none>
kube-system privileged <none>
rutas-norte-dev <none> <none>
rutas-norte-pre <none> restricted
rutas-norte-pro <none> restrictedThe <none> values in enforce are namespaces where anything at all can be deployed right now, default included. That table makes an excellent slide for a security meeting.
Phase 2: enforce: baseline
baseline blocks the genuinely dangerous things —privileged pods, host namespaces, hostPath— and hardly any ordinary application breaches it. It is a step with very little risk and a lot of value.
# k8s/environments/pro/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: rutas-norte-pro
labels:
app.kubernetes.io/part-of: rutas-norte
environment: pro
pod-security.kubernetes.io/enforce: baseline # blocks the worst
pod-security.kubernetes.io/enforce-version: v1.30
pod-security.kubernetes.io/audit: restricted # still reports against the target
pod-security.kubernetes.io/audit-version: v1.30
pod-security.kubernetes.io/warn: restricted
pod-security.kubernetes.io/warn-version: v1.30Now, if somebody tries to deploy a privileged pod:
Error from server (Forbidden): pods "test" is forbidden: violates PodSecurity
"baseline:v1.30": privileged (container "test" must not set securityContext.privileged=true)That is the moment when the work of 08-02 stops depending on anybody's memory.
Phase 3: enforce: restricted and the exact rejection message
Before taking this step, every component must comply. Let us first try an unhardened manifest to see the full message:
# /tmp/unhardened-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: restricted-test
namespace: rutas-norte-pro
spec:
containers:
- name: app
image: registry.rutasnorte.example/utilities:1.4.2kubectl label namespace rutas-norte-pro \
pod-security.kubernetes.io/enforce=restricted \
pod-security.kubernetes.io/enforce-version=v1.30 --overwrite
kubectl apply -f /tmp/unhardened-pod.yamlError from server (Forbidden): error when creating "/tmp/unhardened-pod.yaml":
pods "restricted-test" is forbidden: violates PodSecurity "restricted:v1.30":
allowPrivilegeEscalation != false (container "app" must set
securityContext.allowPrivilegeEscalation=false),
unrestricted capabilities (container "app" must set securityContext.capabilities.drop=["ALL"]),
runAsNonRoot != true (pod or container "app" must set securityContext.runAsNonRoot=true),
seccompProfile (pod or container "app" must set securityContext.seccompProfile.type
to "RuntimeDefault" or "Localhost")The message is a to-do list. We fix it:
# /tmp/hardened-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: restricted-test
namespace: rutas-norte-pro
spec:
securityContext:
runAsNonRoot: true
runAsUser: 10001
seccompProfile:
type: RuntimeDefault
containers:
- name: app
image: registry.rutasnorte.example/utilities:1.4.2
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]The component that did not comply: bookings-postgres
In 08-02 we left readOnlyRootFilesystem: false on PostgreSQL as a documented exception. Good news: restricted does not require that field, so that point does not block. What it does require is runAsNonRoot: true and drop: ["ALL"], which we already had. bookings-postgres passes restricted unchanged.
Let us check before touching the namespace:
No warnings. Perfect.
The component that does not comply: the log collector
Warning: would violate PodSecurity "restricted:v1.30": hostPath volumes (volume "varlog"),
runAsNonRoot != true (pod or container "collector" must set securityContext.runAsNonRoot=true),
restricted volume types (volume "varlog" uses restricted volume type "hostPath")And here there is nothing to fix: the collector legitimately needs to read the node's /var/log and needs to be root in order to do so. That is the subject of the next section.
Final state of the three environments
| Namespace | enforce |
audit / warn |
Reason |
|---|---|---|---|
rutas-norte-dev |
baseline |
restricted |
Developers need room to debug |
rutas-norte-pre |
restricted |
restricted |
It must be identical to production |
rutas-norte-pro |
restricted |
restricted |
The target |
rutas-norte-sistema |
privileged |
— | Node agents (section 6) |
That rutas-norte-pre is just as strict as production is non-negotiable: if pre-production allows what production does not, the promotion fails at the worst possible moment. That is the entire point of having an intermediate environment.
rutas-norte-dev on baseline is a deliberate concession: it blocks the dangerous things but allows, for example, running as root to try something out. With warn: restricted, the developer sees on every apply exactly what will be missing for promotion. It is a balance decision that has to be documented.
- Workloads that legitimately need privileges
There is software that cannot comply with restricted, and not out of laziness. The CNI plugin configures the node's network interfaces. The CSI driver mounts volumes on the host file system. The log collector reads /var/log. The runtime detection agent (Falco, which we will see in 08-06) observes system calls.
The correct answer is not to relax the profile of rutas-norte-pro. It is to isolate those workloads.
Solution 1: a separate namespace with the privileged profile
apiVersion: v1
kind: Namespace
metadata:
name: rutas-norte-sistema
labels:
app.kubernetes.io/part-of: rutas-norte
pod-security.kubernetes.io/enforce: privileged
pod-security.kubernetes.io/enforce-version: v1.30
# audit and warn on restricted: this leaves a trace of everything that skips the profile
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/audit-version: v1.30
pod-security.kubernetes.io/warn: restricted
pod-security.kubernetes.io/warn-version: v1.30
annotations:
security.rutasnorte.example/justification: >-
Namespace for node agents that require host access:
log collector (reading /var/log) and runtime detection
agent. Quarterly review by the platform team
and by security. No business application may be deployed here.Three important decisions in that manifest:
auditandwarnstay onrestricted. Even thoughenforceisprivileged, every pod that skips the profile leaves a trace in the audit log. That way the privileged namespace is not a black hole: we know exactly what runs there and with what privileges.- The justification annotation. A namespace with the
privilegedprofile is a security exception and must be documented on the object itself, with who reviews it and when. - Strict RBAC on top. Remembering 08-01: whoever can create pods here can create privileged pods. Only
platformshould have permission, and in 08-01 we saw that creating pods in a namespace is equivalent to being able to use any ServiceAccount in that namespace: another reason for the applications not to live there.
And the essential RBAC counterpart:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: platform-system
namespace: rutas-norte-sistema
subjects:
- kind: Group
name: platform # ONLY platform. Not development, not CI, not support
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: admin
apiGroup: rbac.authorization.k8s.ioSolution 2: exemptions in the apiserver configuration
PSA accepts global exemptions on three criteria:
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: PodSecurity
configuration:
apiVersion: pod-security.admission.config.k8s.io/v1
kind: PodSecurityConfiguration
defaults:
enforce: "baseline"
enforce-version: "v1.30"
exemptions:
# By user: the subject that creates the pod is exempt
usernames:
- "system:serviceaccount:kube-system:daemon-set-controller"
# By runtime class: pods with this RuntimeClass are exempt
runtimeClasses:
- "gvisor"
# By namespace: nothing in the namespace is evaluated
namespaces:
- "kube-system"
- "rutas-norte-sistema"| Criterion | When to use it | Risk |
|---|---|---|
namespaces |
Whole infrastructure namespaces | Medium: you must watch who can deploy there |
usernames |
One specific controller that creates system pods | High: that identity is exempt across the whole cluster |
runtimeClasses |
Pods that already run sandboxed (gVisor, Kata) | Low: the isolation comes from the runtime |
A warning about usernames: it exempts the subject in every namespace, not just where you want it. It is a very broad permission. The per-namespace exemption is easier to reason about and to audit.
And the general warning: every exemption is a security exception. It must have a written justification, an owner, and a periodic review confirming that it is still needed. At Rutas Norte the exemption list is reviewed every quarter along with RBAC.
- The limitations of Pod Security Admission
PSA is excellent at what it does, and what it does is quite narrow. Knowing its limits saves you from trusting it for things it does not do.
Limitation 1: it only looks at the pod
PSA evaluates Pod objects. Full stop. It does not look at Services, or Ingresses, or PVCs, or ConfigMaps, or NetworkPolicies.
This has an uncomfortable practical effect with enforce:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreate 12s replicaset-controller Error creating: pods
"bad-application-6c5f9b8d4-" is forbidden: violates PodSecurity "restricted:v1.30":
allowPrivilegeEscalation != false (container "app" must set
securityContext.allowPrivilegeEscalation=false), ...The Deployment is created successfully, the ReplicaSet is created, and it is the ReplicaSet that fails to create pods. Whoever ran the apply sees a success and has to go and look at the events to discover the problem.
That is why warn is so valuable: it does evaluate the objects that create pods and warns at the moment of the apply. Always keep warn enabled even when you have enforce, precisely for this reason.
Limitation 2: it cannot require anything outside the securityContext
Things PSA cannot require:
| It cannot require | Why it matters at Rutas Norte |
|---|---|
Labels (app.kubernetes.io/part-of) |
Without them the selectors and the module 7 dashboards break |
resources.requests and limits |
Without them the QoS is BestEffort (03-05) and the pod is the first to be evicted |
| A specific image registry | Anybody can deploy from Docker Hub without review |
That the image does not use latest |
Irreproducible deployments (08-05) |
Health probes (liveness, readiness) |
Without them nothing from module 7 works |
readOnlyRootFilesystem |
Not even restricted requires it |
| That a NetworkPolicy exists | The microsegmentation of 08-04 |
automountServiceAccountToken: false |
What we worked on in 03-06 |
That list is exactly the gap Kyverno and Gatekeeper fill.
Limitation 3: the three profiles are not configurable
You cannot create a profile that is "restricted but allowing read-only hostPath" or "restricted plus readOnlyRootFilesystem". The profiles are defined by the Kubernetes project and they are immutable.
Limitation 4: it does not mutate
It is a design virtue (it avoids PSP's problem 3), but it means PSA will never help you by filling in fields. If you want every pod to get seccompProfile: RuntimeDefault automatically, you need a mutating policy engine.
Summary
| Need | Tool |
|---|---|
Prevent privileged pods, hostPath, root |
PSA (built in, nothing to install) |
| Require labels, limits, an image registry | Kyverno or Gatekeeper |
| Fill in fields automatically | Kyverno (mutation) |
| Verify image signatures | Kyverno or the Sigstore policy-controller (08-05) |
| Simple rules with nothing to install | ValidatingAdmissionPolicy (section 10) |
The practical recommendation: PSA always, as a baseline that cannot fail (it lives in the apiserver and depends on no pod), and a policy engine on top for everything else.
- General policy engines: Kyverno and OPA Gatekeeper
A policy engine is an admission webhook: the apiserver sends it every object and it answers whether it accepts, rejects or modifies it.
flowchart LR
A["kubectl apply"] --> B["apiserver"]
B --> C["PSA<br/>(built in)"]
C --> D["Mutating webhook<br/>Kyverno"]
D --> E["Validating webhook<br/>Kyverno / Gatekeeper"]
E -->|Rejects| X["Error with the reason"]
E -->|Accepts| F[("etcd")]
This brings a very serious operational consideration: the policy engine becomes a dependency of the apiserver. If the webhook is configured with failurePolicy: Fail and the engine's pods are down, nobody can create anything in the cluster. Whole clusters have gone down that way.
failurePolicy |
If the webhook does not respond | When to use it |
|---|---|---|
Ignore |
The object is accepted | Initial adoption, non-critical policies |
Fail |
The object is rejected | Security policies, with a highly available engine |
The usual mitigations: several engine replicas with anti-affinity (06-05), a PodDisruptionBudget (09-05), and excluding kube-system from the webhooks so that the cluster can be recovered.
Comparison: Kyverno and OPA Gatekeeper
| Kyverno | OPA Gatekeeper | |
|---|---|---|
| Policy language | YAML (like Kubernetes) | Rego (OPA's own language) |
| Learning curve | Low: if you know YAML, you can write policies | High: Rego is a different declarative language |
| Validate | Yes | Yes |
| Mutate | Yes, very well resolved | Yes, more limited |
| Generate objects | Yes (create a NetworkPolicy with each namespace) | No |
| Verify image signatures | Yes, integrated with Cosign | Requires additional work |
| Resource cleanup | Yes (CleanupPolicy) |
No |
| Compliance reports | PolicyReport as a CRD |
Violation objects |
| Scope | Kubernetes only | Generic: also CI, Terraform, APIs |
| Ready-made policies | A broad, maintained catalogue | A template library |
| Community and maturity | CNCF, fast growth | CNCF, more established |
| Performance with many policies | Good | Very good |
Recommendation for Rutas Norte: Kyverno. The reasons are concrete:
- The team already writes YAML all day. Learning Rego would be a real barrier to developers writing policies too.
- Signature verification with Cosign is built in, and that is exactly what we will need in 08-05.
- The ability to generate objects allows, for example, automatically creating the
deny-allNetworkPolicy from 04-06 in every new namespace. - Mutation solves the "add
seccompProfileto everything" problem without touching a hundred manifests.
Gatekeeper is the better option if the organisation already uses OPA for other things (API access control, Terraform policies) and wants a single policy language for everything.
Installing Kyverno
helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
helm install kyverno kyverno/kyverno \
--namespace kyverno --create-namespace \
--set admissionController.replicas=3 \
--set admissionController.podDisruptionBudget.minAvailable=2Three replicas and a PDB: the policy engine is critical infrastructure and must be treated as such.
- Kyverno policies for Rutas Norte
A Kyverno policy is a ClusterPolicy (the whole cluster) or a Policy (one namespace) with a list of rules.
Policy 1: every image from the company registry
This is probably the highest-value policy per line written. It prevents anything being deployed that has not gone through the company registry, where in 08-05 and 08-06 we will put the scanning and the signing.
# k8s/policies/mandatory-registry.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: mandatory-registry
annotations:
policies.kyverno.io/title: Mandatory image registry
policies.kyverno.io/category: Supply chain security
policies.kyverno.io/severity: high
policies.kyverno.io/description: >-
Every image deployed in the Rutas Norte namespaces must come from
registry.rutasnorte.example. Public images are mirrored there
beforehand, where they are scanned and signed. See lessons 08-05 and 08-06.
spec:
# Fail: if Kyverno cannot evaluate, it rejects. This is a security policy.
validationFailureAction: Enforce
background: true # also evaluates existing objects, for the reports
rules:
- name: check-registry
match:
any:
- resources:
kinds:
- Pod
namespaces:
- rutas-norte-dev
- rutas-norte-pre
- rutas-norte-pro
validate:
message: >-
The image "{{ request.object.spec.containers[0].image }}" does not come from
registry.rutasnorte.example. Every image must be mirrored into the
company registry, where images are scanned and signed.
See the platform image guide.
pattern:
spec:
# The = sign means "if the field exists, it must comply"
=(ephemeralContainers):
- image: "registry.rutasnorte.example/*"
=(initContainers):
- image: "registry.rutasnorte.example/*"
containers:
- image: "registry.rutasnorte.example/*"Let us take the manifest apart:
validationFailureAction: Enforcerejects. The alternative isAudit, which only produces a report. Just as with PSA, you start onAuditand move toEnforcewhen there are no breaches left.background: truemakes Kyverno also evaluate objects that already exist, producingPolicyReportobjects. Without it you would only see new objects.match.any.resourcesdefines the scope. Here, pods in the three Rutas Norte namespaces.kyverno,kube-systemandrutas-norte-sistemaare left out because they use external infrastructure images.=(initContainers)and=(ephemeralContainers): the=()prefix means "if this field exists, it must match the pattern". Without it, a pod with no initContainers would fail validation. Do not forget the ephemeral containers: they are thekubectl debugroute (07-06) and without this line somebody could inject an arbitrary image into a production pod.- The message appears verbatim in the terminal of whoever is deploying. Writing it well —what fails, why, and what to do— saves an enormous number of questions.
Trying it out:
Error from server: admission webhook "validate.kyverno.svc-fail" denied the request:
resource Pod/rutas-norte-pro/test was blocked due to the following policies
mandatory-registry:
check-registry: 'validation error: The image "nginx:latest" does not come from
registry.rutasnorte.example. Every image must be mirrored into the company
registry, where images are scanned and signed. See the platform image
guide. rule check-registry failed at path /spec/containers/0/image/'Policy 2: require the labels of the official scheme
In 02-07 we defined the Rutas Norte label scheme, and in module 7 the dashboards and the alerts depend on it. A workload without app.kubernetes.io/part-of: rutas-norte is invisible to monitoring.
# k8s/policies/mandatory-labels.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: mandatory-labels
annotations:
policies.kyverno.io/title: Rutas Norte label scheme
policies.kyverno.io/category: Governance
policies.kyverno.io/severity: medium
spec:
validationFailureAction: Enforce
background: true
rules:
- name: labels-on-workloads
match:
any:
- resources:
kinds:
- Deployment
- StatefulSet
- DaemonSet
- CronJob
- Job
namespaces:
- rutas-norte-dev
- rutas-norte-pre
- rutas-norte-pro
validate:
message: >-
Mandatory labels are missing. Every Rutas Norte workload must carry,
in metadata and in the pod template: "app" (the component name),
"app.kubernetes.io/part-of: rutas-norte" and "environment" (dev|pre|pro).
Without them, the workload appears in neither the dashboards nor the alerts.
pattern:
metadata:
labels:
app: "?*" # ?* = not empty
app.kubernetes.io/part-of: "rutas-norte" # exact value
environment: "dev | pre | pro" # one of the three
spec:
template:
metadata:
labels:
app: "?*"
app.kubernetes.io/part-of: "rutas-norte"
environment: "dev | pre | pro"
- name: environment-namespace-consistency
match:
any:
- resources:
kinds: [Deployment, StatefulSet, DaemonSet, CronJob]
namespaces: [rutas-norte-pro]
validate:
message: >-
In rutas-norte-pro the "environment" label must be "pro".
An incorrect label makes the production alerts get
routed to the wrong channel (see 07-04).
pattern:
metadata:
labels:
environment: "pro"The Kyverno pattern syntax that appears here:
| Syntax | Meaning |
|---|---|
"?*" |
Any non-empty value |
"rutas-norte" |
That exact value |
"dev | pre | pro" |
One of those three (the | is an "or") |
"registry.rutasnorte.example/*" |
A trailing wildcard |
=(field) |
If the field exists, it must comply |
X(field) |
The field must not exist |
The second rule is a detail that looks minor and is not: an environment: dev label on a manifest copied into production makes Alertmanager route the alerts to the development channel and nobody hears about a real outage. The policy prevents it.
Policy 3: require resource limits
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: mandatory-resources
spec:
validationFailureAction: Enforce
background: true
rules:
- name: requests-and-limits
match:
any:
- resources:
kinds: [Pod]
namespaces: [rutas-norte-pre, rutas-norte-pro]
validate:
message: >-
Every container must declare CPU and memory requests and limits.
Without them the QoS class is BestEffort (see 03-05) and the pod is the
first to be evicted when the node is short of resources.
pattern:
spec:
containers:
- resources:
requests:
cpu: "?*"
memory: "?*"
limits:
memory: "?*"Notice that we require limits.memory but not limits.cpu. It is deliberate and it is a debated decision: limiting the CPU causes throttling that can degrade bookings-api latency without it being obvious why, whereas not limiting memory can take the node down. It is a good example of a policy that codifies a technical decision made by the team; we will cover it thoroughly in 09-06.
Policy 4: mutate to add the security baseline
This is where Kyverno shines compared with PSA:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: add-security-baseline
spec:
rules:
- name: default-seccomp
match:
any:
- resources:
kinds: [Pod]
namespaces: [rutas-norte-dev, rutas-norte-pre, rutas-norte-pro]
mutate:
patchStrategicMerge:
spec:
# +() means "add this value ONLY if the field does not exist"
+(securityContext):
seccompProfile:
type: RuntimeDefaultWith +(), if the manifest already defines securityContext, Kyverno leaves it alone; if it does not, it adds it. It is a safety net, not a replacement for the work of 08-02: the right thing is still to declare it explicitly in the manifest, so that whoever reads it knows what is running.
A warning about mutating: it is exactly problem 3 of PodSecurityPolicy. Use it sparingly and only for safe defaults, never to "fix" incorrect manifests, because then the repository's YAML stops describing what runs.
Policy 5: generate the deny-all NetworkPolicy in every new namespace
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: generate-deny-all
spec:
rules:
- name: deny-all-in-rutas-norte-namespaces
match:
any:
- resources:
kinds: [Namespace]
selector:
matchLabels:
app.kubernetes.io/part-of: rutas-norte
generate:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
name: deny-all
namespace: "{{request.object.metadata.name}}"
synchronize: true # if somebody deletes it, Kyverno recreates it
data:
spec:
podSelector: {}
policyTypes: [Ingress, Egress]This guarantees that no Rutas Norte namespace can exist without the default deny we built in 04-06. And synchronize: true means that, if somebody deletes it, it comes back on its own. It is a capability Gatekeeper does not have and that solves a real problem.
Consulting the reports
kubectl get policyreport -n rutas-norte-pro -o json | jq -r '
.items[].results[] | select(.result == "fail")
| "\(.policy)/\(.rule): \(.resources[0].kind)/\(.resources[0].name)\n \(.message)"'mandatory-labels/labels-on-workloads: Deployment/legacy-exporter
validation error: Mandatory labels are missing...
mandatory-resources/requests-and-limits: Pod/temp-debug-x8k2p
validation error: Every container must declare requests and limits...Those reports are the outstanding to-do list and, with background: true, they keep themselves up to date. In 08-06 we will turn them into evidence for compliance audits.
- ValidatingAdmissionPolicy: native CEL with nothing to install
Since Kubernetes 1.30, the ValidatingAdmissionPolicy lets you write admission rules inside the apiserver, with no webhook, using CEL (Common Expression Language) expressions.
Advantages over an external engine:
| ValidatingAdmissionPolicy | Webhook (Kyverno/Gatekeeper) | |
|---|---|---|
| Installation | None | Helm, pods, certificates |
| Availability | That of the apiserver | It can go down and block the cluster |
| Latency | No network hop | One HTTP call per object |
| Mutation | No (there is an alpha MutatingAdmissionPolicy) |
Yes |
| Generate objects | No | Kyverno does |
| Verify signatures | No | Kyverno does |
| Expressiveness | CEL: good for field rules | Very high |
Example: forbid the latest tag
It is made up of two objects: the policy (what is checked) and the binding (where it applies).
# k8s/policies/vap-forbid-latest.yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: forbid-latest-tag
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- apiGroups: [""]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["pods"]
validations:
- expression: >-
object.spec.containers.all(c,
!c.image.endsWith(":latest") && c.image.contains(":"))
message: >-
Images may not use the "latest" tag and must carry an explicit
tag. Use an immutable tag or, better still, a digest
(image@sha256:...). See lesson 08-05.
reason: Invalid
- expression: >-
!has(object.spec.initContainers) ||
object.spec.initContainers.all(c,
!c.image.endsWith(":latest") && c.image.contains(":"))
message: 'The same rule applies to the initContainers.'
reason: Invalid
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: forbid-latest-tag
spec:
policyName: forbid-latest-tag
validationActions: [Deny] # Deny | Warn | Audit, combinable
matchResources:
namespaceSelector:
matchLabels:
app.kubernetes.io/part-of: rutas-nortePoints to note:
validationActionsplays the same role as the PSA modes:[Warn, Audit]to try it out,[Deny]when you are sure. They can be combined:[Deny, Audit].namespaceSelectorapplies the policy only to the namespaces labelled as part of Rutas Norte. The system ones are left out.- CEL has very readable functions:
all(),exists(),has(),endsWith(),contains(),matches()for regular expressions. - Separating policy and binding lets you write the rule once and apply it with different severity in different environments:
[Warn]indevand[Deny]inpro, with two bindings.
A test:
The pods "test" is invalid: ValidatingAdmissionPolicy 'forbid-latest-tag'
with binding 'forbid-latest-tag' denied request: Images may not use
the "latest" tag and must carry an explicit tag. Use an immutable tag
or, better still, a digest (image@sha256:...). See lesson 08-05.An example with parameters
VAPs can read configuration from an external resource, which makes them reusable:
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: limit-replicas
spec:
failurePolicy: Fail
paramKind:
apiVersion: v1
kind: ConfigMap
matchConstraints:
resourceRules:
- apiGroups: ["apps"]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["deployments"]
validations:
- expression: >-
object.spec.replicas <= int(params.data.maxReplicas)
messageExpression: >-
"The number of replicas (" + string(object.spec.replicas) +
") exceeds the maximum allowed (" + params.data.maxReplicas + ")."messageExpression builds the message with the real values, something you appreciate a great deal when debugging.
Criteria for use
Use VAP for simple rules on fields: forbidding latest, requiring a label, capping a numeric value, checking a prefix. They are free in availability terms and add no piece to maintain.
Use Kyverno for what VAP cannot do: verifying image signatures (08-05), generating objects, mutating, or policies that need to look at other cluster resources.
And always use PSA as the securityContext baseline, because it is built in and cannot fail.
At Rutas Norte we use all three, in layers:
flowchart TD
A["Every pod that is created"] --> B["PSA: restricted<br/>securityContext baseline<br/>(built in, infallible)"]
B --> C["VAP: simple rules<br/>no latest, mandatory fields<br/>(built in)"]
C --> D["Kyverno: registry, signatures,<br/>labels, resources, generation<br/>(webhook)"]
D --> E["Pod admitted"]
style B fill:#d5f9d5,stroke:#3a3
style C fill:#d5e8f9,stroke:#36c
style D fill:#f9f0d5,stroke:#ca3
Common Mistakes and Tips
Looking for PodSecurityPolicy tutorials. They were removed in 1.25. If a resource talks to you about kind: PodSecurityPolicy, it is out of date and probably everything else in it is too.
Enabling enforce: restricted straight away in production. The existing pods carry on running, but the first restart or the first update fails. Always walk the phases: warn/audit, then baseline, then restricted.
Not pinning -version in the labels. Without it latest is used and a cluster upgrade can reject pods that used to pass. Pin the version and raise it as an explicit change.
Believing that enforce warns when you apply a Deployment. It does not: PSA only evaluates pods. The Deployment is created and it is the ReplicaSet that fails, in an event you have to go looking for. Always keep warn enabled, since it does evaluate the objects that create pods.
Leaving namespaces unlabelled. A namespace with no PSA labels is equivalent to privileged. Check default and any namespace created by hand; configure the default in the apiserver or require the labels with Kyverno.
Relaxing a whole namespace's profile because of one workload. If the log collector needs hostPath, it goes to rutas-norte-sistema with the privileged profile; you do not drop rutas-norte-pro to baseline.
Forgetting the RBAC of the privileged namespace. A namespace with the privileged profile where development can create pods is worse than having no PSA: it gives a false sense of security. And remember from 08-01 that creating pods there allows using any ServiceAccount in that namespace.
Using failurePolicy: Fail without a highly available engine. If Kyverno goes down, nobody can create anything. Three replicas, a PDB and anti-affinity as a minimum, and exclude kube-system.
Forgetting initContainers and ephemeralContainers in the policies. An initContainer with an unverified image bypasses the entire registry policy. Ephemeral containers are the kubectl debug route.
Setting validationFailureAction: Enforce from day one. Just as with PSA: Audit first, look at the PolicyReport objects, fix, and only then Enforce.
Overusing mutation. If Kyverno "fixes" the manifests, the repository's YAML stops describing what runs. Use it only for safe defaults.
Useless error messages. The message is read by a person who is trying to deploy. Tell them what fails, why the rule exists and what they have to do. A good message saves the platform team an interruption.
Golden tip: policies are code. They live in k8s/policies/ in the repository, they are reviewed in a pull request, and they are tested in rutas-norte-dev before reaching production. A policy applied by hand in production is exactly the same problem as a RoleBinding applied by hand.
Exercises
Exercise 1: prepare a namespace for restricted
The team creates rutas-norte-analytics to deploy an occupancy statistics aggregation service. Write:
- The
Namespacemanifest in phase 1 (warn without blocking) with the version pinned. - The command that would evaluate an existing manifest against the profile without deploying it.
- The
Namespacemanifest in its final state, given that the service is already hardened.
Exercise 2: interpret and fix a rejection
Deploying a new component in rutas-norte-pro, which has enforce: restricted, you get:
Error from server (Forbidden): error when creating "exporter.yaml":
pods "timetable-exporter" is forbidden: violates PodSecurity "restricted:v1.30":
non-default capabilities (container "exporter" must not include "NET_RAW" in
securityContext.capabilities.add), restricted volume types (volume "node-config"
uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container
"exporter" must set securityContext.runAsNonRoot=true), seccompProfile
(pod or container "exporter" must set securityContext.seccompProfile.type
to "RuntimeDefault" or "Localhost")The original manifest:
apiVersion: v1
kind: Pod
metadata:
name: timetable-exporter
namespace: rutas-norte-pro
spec:
containers:
- name: exporter
image: registry.rutasnorte.example/timetable-exporter:2.1.0
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
add: ["NET_RAW"]
volumeMounts:
- name: node-config
mountPath: /etc/timetables
volumes:
- name: node-config
hostPath:
path: /etc/rutasnorte/timetables
type: Directory- List the four violations and explain what each one means.
- Rewrite the manifest so that it complies with
restricted, knowing thatNET_RAWwas added "just in case" and that the file in/etc/rutasnorte/timetablesis in fact a static configuration file. - What would you have done if
NET_RAWreally had been indispensable?
Exercise 3: a Kyverno policy for automountServiceAccountToken
In 03-06 we established that workloads which do not talk to the API must carry automountServiceAccountToken: false. Write a Kyverno ClusterPolicy that:
- Applies to Pods in
rutas-norte-preandrutas-norte-pro. - Rejects pods that do not declare
automountServiceAccountTokenexplicitly (neithertruenorfalse), forcing it to be a conscious decision. - Exempts pods whose ServiceAccount is
bookings-apiorinventory-agent, which do need the token. - Includes a message explaining what to do.
Also state how you would roll it out safely.
Solutions
Solution 1
1. Phase 1: warn without blocking
# k8s/environments/analytics/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: rutas-norte-analytics
labels:
app.kubernetes.io/part-of: rutas-norte
environment: pro
# Phase 1: no enforce. We are only observing.
pod-security.kubernetes.io/warn: restricted
pod-security.kubernetes.io/warn-version: v1.30
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/audit-version: v1.30
annotations:
security.rutasnorte.example/psa-phase: >-
Phase 1 (warn+audit) since 2026-08-06. Target: enforce:restricted
before 2026-09-01. Owner: the platform team.The annotation with a target date stops "temporary" from becoming "permanent", which is how almost every phase 1 ends up.
2. Evaluate without deploying
Warning: would violate PodSecurity "restricted:v1.30": allowPrivilegeEscalation != false
(container "aggregator" must set securityContext.allowPrivilegeEscalation=false),
unrestricted capabilities (container "aggregator" must set
securityContext.capabilities.drop=["ALL"])
deployment.apps/aggregator created (server dry run)--dry-run=server passes the object through the whole admission chain —PSA, VAP and Kyverno included— and returns the verdict without storing anything. It is different from --dry-run=client, which only validates the YAML locally and would detect none of this.
3. Final state
apiVersion: v1
kind: Namespace
metadata:
name: rutas-norte-analytics
labels:
app.kubernetes.io/part-of: rutas-norte
environment: pro
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/enforce-version: v1.30
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/audit-version: v1.30
pod-security.kubernetes.io/warn: restricted
pod-security.kubernetes.io/warn-version: v1.30audit and warn are kept even though enforce already blocks: warn still evaluates the Deployments (which enforce does not see) and audit leaves a record in the audit log.
Solution 2
1. The four violations:
| Violation | Meaning |
|---|---|
non-default capabilities: NET_RAW |
restricted only allows adding NET_BIND_SERVICE back. NET_RAW (raw sockets) is forbidden |
restricted volume types: hostPath |
hostPath is not on the list of volumes restricted allows. It is the formalisation of the warning from 05-01 |
runAsNonRoot != true |
It was not declared; restricted requires the explicit declaration, it is not enough for the image to have a USER |
seccompProfile |
RuntimeDefault is missing. Remember from 08-02 that without declaring it there is probably no filter at all |
2. Corrected manifest:
# k8s/environments/pro/timetable-exporter.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: timetables-config
namespace: rutas-norte-pro
labels:
app: timetable-exporter
app.kubernetes.io/part-of: rutas-norte
environment: pro
data:
timetables.yaml: |
routes:
- code: N-101
origin: Bilbao
destination: Santander
departures: ["07:00", "10:30", "15:00", "19:45"]
---
apiVersion: v1
kind: Pod
metadata:
name: timetable-exporter
namespace: rutas-norte-pro
labels:
app: timetable-exporter
app.kubernetes.io/part-of: rutas-norte
environment: pro
spec:
automountServiceAccountToken: false
securityContext:
runAsNonRoot: true # violation 3
runAsUser: 10005
runAsGroup: 10005
seccompProfile:
type: RuntimeDefault # violation 4
containers:
- name: exporter
image: registry.rutasnorte.example/timetable-exporter:2.1.0
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true # restricted does not require it, but it is right
capabilities:
drop: ["ALL"] # violation 1: without the NET_RAW add
volumeMounts:
- name: timetables # violation 2: a ConfigMap instead of hostPath
mountPath: /etc/timetables
readOnly: true
- name: temp
mountPath: /tmp
resources:
requests: { cpu: 50m, memory: 64Mi }
limits: { memory: 128Mi }
volumes:
- name: timetables
configMap:
name: timetables-config
- name: temp
emptyDir: { sizeLimit: 32Mi }The four changes and their justification:
NET_RAWremoved. It was "just in case": it allows creating raw sockets, which is useful forpingand for inspecting traffic. A timetable exporter does not need it. If the process fails to start, the log will say so and then we will investigate why; capabilities are not granted pre-emptively.hostPathreplaced with aConfigMap. The file was static configuration, which is exactly what a ConfigMap is for (03-01). Besides satisfying the policy, this is a gain: the file no longer depends on somebody having copied it onto every node, it is versioned in Git and it is identical in all three environments. Fixing the policy breach improved the design, which is what usually happens.runAsNonRoot: true+runAsUser: 10005declared explicitly.seccompProfile: RuntimeDefaulton the pod, inherited by the container.
Extras that restricted did not require but the Kyverno policies from section 9 do: the scheme labels, resources and automountServiceAccountToken: false. Plus readOnlyRootFilesystem: true, which no policy requires but which is the baseline from 08-02.
3. If NET_RAW really were indispensable:
The process would be this, in order:
- Question the requirement. Why does a timetable exporter need raw sockets? In 90% of cases there is an alternative (a TCP check instead of ICMP, for instance).
- If it is real, use a library that does not need it or change the design.
- If there is no alternative, do not relax
rutas-norte-pro. It gets deployed in its own namespace withenforce: baseline(which does allowNET_RAW), with restricted RBAC, with an annotated justification, with a quarterly review and with approval from the security professional. - Document the exception in the security exception register, with an expiry date. We will come back to this register in 08-06.
What you never do is drop rutas-norte-pro to baseline: that turns one component's exception into an exception for the whole platform.
Solution 3
# k8s/policies/explicit-serviceaccount-token.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: explicit-serviceaccount-token
annotations:
policies.kyverno.io/title: Explicit decision on the ServiceAccount token
policies.kyverno.io/category: Security
policies.kyverno.io/severity: medium
policies.kyverno.io/description: >-
Every pod must declare automountServiceAccountToken explicitly.
Mounting the token without needing it hands an API credential to a
process that does not use it (see 03-06). The workloads that do talk to the API
are exempted by ServiceAccount name.
spec:
validationFailureAction: Enforce
background: true
rules:
- name: require-explicit-declaration
match:
any:
- resources:
kinds: [Pod]
namespaces:
- rutas-norte-pre
- rutas-norte-pro
exclude:
any:
# Workloads that legitimately need the token
- resources:
subjects:
- kind: ServiceAccount
name: bookings-api
- kind: ServiceAccount
name: inventory-agent
preconditions:
all:
# Do not apply to the pods the system itself creates
- key: "{{ request.object.metadata.namespace }}"
operator: AnyIn
value: ["rutas-norte-pre", "rutas-norte-pro"]
validate:
message: >-
The pod must declare spec.automountServiceAccountToken
explicitly. If the workload does NOT talk to the Kubernetes API (the usual case),
set "automountServiceAccountToken: false". If it does, set it to true
and make sure its ServiceAccount has a Role with least
privilege (see 08-01 and 03-06).
pattern:
spec:
automountServiceAccountToken: "false | true"Notes on the solution:
- The
"false | true"pattern forces the field to exist with one of those two values. If the field is absent, validation fails, which is exactly what the exercise asked for: turning an omission into a decision. exclude.any.resources.subjectsallows exempting by ServiceAccount. A more maintainable alternative in the long run would be to exempt by a label (security.rutasnorte.example/uses-api: "true"), so that the policy does not have to be edited every time a new workload needs the API.- It is a governance policy, not an attack-blocking one: its value lies in forcing somebody to think on every deployment.
Safe rollout, in four steps:
# 1. Apply in Audit mode: it blocks nothing
sed 's/validationFailureAction: Enforce/validationFailureAction: Audit/' \
k8s/policies/explicit-serviceaccount-token.yaml | kubectl apply -f -
# 2. Wait for the background scan to complete the report and review it
kubectl get policyreport -A -o json | jq -r '
.items[].results[]
| select(.policy == "explicit-serviceaccount-token" and .result == "fail")
| "\(.resources[0].namespace)/\(.resources[0].name)"'rutas-norte-pro/notifications-worker-6d4f8b9c7-k2m4x
rutas-norte-pro/occupancy-reports-29187360-9wzqt
rutas-norte-pre/web-store-7c8d9f6b5-p3n8v# 3. Fix the manifests of those workloads (add the explicit field)
# and check that the report comes back clean.
# 4. Only then, move to Enforce
kubectl apply -f k8s/policies/explicit-serviceaccount-token.yamlAnd the final check:
Error from server: admission webhook "validate.kyverno.svc-fail" denied the request:
resource Pod/rutas-norte-pro/test was blocked due to the following policies
explicit-serviceaccount-token:
require-explicit-declaration: 'validation error: The pod must declare
spec.automountServiceAccountToken explicitly...'One additional and important precaution: before moving to Enforce, check that the policy does not block the system controllers that create pods in those namespaces (the ReplicaSet controller, the Job controller). If a controller creates pods without the field, your deployments will stop working. Testing it first in rutas-norte-dev with Enforce for a few days is the cheap way to find out.
Conclusion
We have turned the hardening of 08-02 into something the cluster enforces on its own:
- Admission control is the third gate: RBAC decides whether you have the right to the operation, admission decides whether the object is acceptable. The rejected object never reaches etcd.
- PodSecurityPolicy is history: deprecated in 1.21, removed in 1.25. Its three defects —counter-intuitive RBAC, unpredictable alphabetical ordering and mutation— explain the design of its replacement.
- The Pod Security Standards define three profiles:
privileged(no restrictions),baseline(blocks the known escalations) andrestricted(the target for every application), which is essentially the official codification of what we did by hand in 08-02. - Pod Security Admission is built into the apiserver and is enabled with namespace labels, with three modes —
enforce,audit,warn— that allow gradual adoption. Always pin-versionso that a cluster upgrade breaks nothing. - The correct adoption is phased:
warn+audit, thenenforce: baseline, thenenforce: restricted. Never in one go in production. Andrutas-norte-premust be as strict asrutas-norte-pro. - Workloads that legitimately need privileges —CNI, CSI, the log collector— go into a separate namespace with the
privilegedprofile, restricted RBAC and an annotated justification. You never relax the production profile because of one workload. - PSA has clear limits: it only looks at the pod, it cannot require labels, resource limits or image registries, and it does not mutate.
- That gap is filled by Kyverno (YAML, mutates, generates, verifies signatures: Rutas Norte's choice) and OPA Gatekeeper (Rego, generic beyond Kubernetes), and the ValidatingAdmissionPolicy objects with CEL cover the simple rules with nothing to install.
- Rutas Norte uses all three in layers: PSA as the infallible base, VAP for simple rules, Kyverno for the mandatory registry, the scheme labels, the resource limits and the automatic generation of the
deny-allNetworkPolicy.
The cluster now rejects, on its own, a privileged pod, one that mounts hostPath, one that runs as root or one whose image does not come from the company registry. Nobody can slip past it by oversight.
But let us look at the whole picture again. We have protected who can do what (08-01) and what a container can do (08-02 and 08-03). An entire dimension is missing: what can talk to what. In 04-06 we put a deny-all NetworkPolicy in rutas-norte-pro and authorised the conversations one at a time, and even then we pointed out two uncomfortable limits: the policies work at L3/L4 —they know nothing about HTTP paths or methods— and they log nothing, so a denied connection attempt is invisible. What is more, inside the cluster all the traffic travels unencrypted once past the Ingress TLS: whoever can observe the node's network sees the queries to bookings-postgres in the clear, personal data included. And outbound traffic is still practically unrestricted, which is the natural route by which a customer database gets exfiltrated.
The next lesson, 08-04, Network Security, builds the complete strategy on top of what you already know: microsegmentation, egress traffic control and the problem of IPs versus domain names, encryption in transit with mTLS and service meshes —Istio, Linkerd and Cilium compared, with an honest criterion for when you need one and when you do not—, protecting the perimeter and the control plane, and the traffic visibility that is exactly what NetworkPolicy lacks.
Kubernetes Course
Module 1: Introduction to Kubernetes
- What Is Kubernetes?
- Kubernetes Architecture
- Key Concepts and Terminology
- Setting Up a Kubernetes Cluster
- The Kubernetes CLI: kubectl
- Objects, YAML Manifests and the Declarative Model
- The Course Project: the Rutas Norte Platform
Module 2: Core Kubernetes Components
- Pods
- ReplicaSets
- Deployments
- Updates, Rollbacks and Deployment Strategies
- Services
- Namespaces
- Labels, Selectors and Annotations
Module 3: Configuration and Secret Management
- ConfigMaps
- Secrets
- Environment Variables
- Resource Quotas and Limits
- LimitRanges and Quality of Service (QoS) Classes
- ServiceAccounts and API Access from Pods
Module 4: Networking in Kubernetes
- Cluster Networking
- Service Types
- Internal DNS and Service Discovery
- Ingress Controllers
- TLS and Certificate Management with cert-manager
- Network Policies
Module 5: Storage in Kubernetes
- Volumes
- Persistent Volumes
- Persistent Volume Claims
- Storage Classes
- Dynamic Provisioning, Expansion and Snapshots
- Backup and Restore of Persistent Data
Module 6: Advanced Kubernetes Concepts
- StatefulSets
- DaemonSets
- Jobs and CronJobs
- Init Containers, Sidecars and Multi-Container Patterns
- Scheduling: Affinity, Taints and Tolerations
- Custom Resource Definitions (CRDs)
- Operators and the Controller Pattern
Module 7: Monitoring and Logging
- Health Checks and Probes
- Metrics Server and kubectl top
- Monitoring with Prometheus
- Visualization and Alerting with Grafana and Alertmanager
- Centralized Logging with Elasticsearch, Fluentd and Kibana (EFK)
- Application Debugging and Cluster Events
Module 8: Kubernetes Security
- Role-Based Access Control (RBAC)
- Security Contexts and Container Hardening
- Pod Security Policies and Pod Security Standards
- Network Security
- Image Security
- Auditing, Scanning and Vulnerability Management
Module 9: Scaling and Performance
- Horizontal Pod Autoscaling
- Vertical Pod Autoscaling
- Cluster Autoscaling
- Event-Driven and Custom-Metric Scaling with KEDA
- High Availability: PodDisruptionBudgets and Topology
- Performance Tuning
Module 10: Kubernetes Ecosystem and Tooling
- Minikube and Local Environments with kind
- Kubeadm
- Helm
- Kustomize
- GitOps with Argo CD and Flux
- Managed Kubernetes: EKS, AKS and GKE
Module 11: Case Studies and Real-World Applications
- Deploying a Web Application
- Running Stateful Applications
- CI/CD with Kubernetes
- Deployment Strategies: Blue-Green and Canary
- Multi-Cluster Management
- Production Operations: Incidents, Runbooks and Costs
