In the previous lesson we closed the door on who can do what: RBAC decides whether you have the right to create a pod in rutas-norte-pro. But we finished with a very specific discomfort. Someone from platform, with perfectly legitimate permissions, can deploy a container today with privileged: true, mounting the node's disk with hostPath and running as root. RBAC will say yes, because they have permission to create pods. And from there the whole node —with every pod running on it, bookings-postgres and its personal data included— is laid bare.

This lesson is about the second barrier: what the process can do once it is running inside the container. It is the difference between "somebody has managed to run code in web-store" —a serious but contained incident— and "somebody has managed to run code in web-store and from there read the customer database" —an incident of another magnitude entirely—.

The tool is called securityContext and it is, alongside RBAC, what delivers the most security value per line of YAML written.

Important warning. This lesson explains the hardening mechanisms and proposes an example configuration. The real hardening of a production platform must be reviewed by a security professional, who knows the organisation's specific threat model. When the system handles personal data —as bookings-postgres does— the design must also be known to the compliance officer. The approach here is exclusively defensive: understanding the mechanisms in order to close them, never to exploit them.

Contents

  1. What a container actually isolates
  2. The securityContext: pod level and container level
  3. Running without root: runAsNonRoot, runAsUser and runAsGroup
  4. Volumes and file permissions: fsGroup and fsGroupChangePolicy
  5. allowPrivilegeEscalation and the no_new_privs bit
  6. privileged: true and why it is the same as handing over the node
  7. Linux capabilities: drop everything and add only what is needed
  8. readOnlyRootFilesystem and how to make it workable
  9. Seccomp, AppArmor and SELinux
  10. The pod settings to avoid
  11. RuntimeClass and sandboxed runtimes
  12. The full hardening of Rutas Norte
  13. How to check the result from inside the container
  14. Common mistakes and tips
  15. Exercises
  16. Conclusion

  1. What a container actually isolates

Before configuring anything you have to understand what you are configuring. And the starting statement is uncomfortable:

A container is not a virtual machine. All the containers on a node share the same Linux kernel. A container is an ordinary process of the node's operating system, restricted in three ways: what it sees, what it consumes and what it can ask the kernel for.

Compare it with a virtual machine:

Virtual machine Container
Kernel Its own, isolated Shared with the node
Isolation boundary Hypervisor (hardware) Kernel features (software)
Attack surface The hypervisor interface (small) Every system call (~350)
Start-up Seconds or minutes Milliseconds
Footprint A complete operating system Just the process
Escape Very difficult Possible with a kernel bug or a misconfiguration

The three mechanisms that provide the isolation:

Kernel namespaces: what the process sees

Not to be confused with Kubernetes Namespace objects: they are a completely different Linux concept. A kernel namespace gives the process its own view of a system resource.

Namespace What it isolates The Kubernetes setting that breaks it
PID The visible processes hostPID: true
Network Interfaces, IP, ports, routing tables hostNetwork: true
Mount The file tree hostPath (partially)
IPC Shared memory, message queues hostIPC: true
UTS Host name
User UID mapping (user namespaces, still optional)

When you run ps aux inside a container and see only your process as PID 1, that is the PID namespace at work. When hostPID: true is enabled, you see every process on the node, including those of the other pods, with their command lines and their environment variables.

cgroups: how much it can consume

Control groups limit CPU, memory, I/O and the number of processes. They are what sits behind the resources.limits of module 3. Their purpose is mainly stability —stopping one pod from taking the node down— but also security, because a local denial of service is an attack.

Capabilities and system-call filters: what it can ask the kernel for

This is where the securityContext makes the difference and where the game is won or lost. We will come back to it in sections 7 and 9.

What this means in practice

The fact that all containers share a kernel has a direct consequence:

  • A Linux kernel bug can allow an escape from the container. Such bugs turn up periodically and get fixed; that is why keeping the nodes patched is a first-order security task, not routine maintenance.
  • The less a container can ask the kernel for, the less surface it has to take advantage of one of those bugs. That is the logic behind everything that follows.
  • If you need strong isolation for something you do not control, ordinary containers are not the answer: you need a sandboxed runtime (section 11) or an outright virtual machine.
flowchart TB
    subgraph Node["Kubernetes node"]
        K["Linux kernel — SHARED"]
        subgraph C1["Pod web-store"]
            P1["nginx<br/>PID, net, mount ns<br/>cgroups<br/>reduced capabilities<br/>seccomp"]
        end
        subgraph C2["Pod bookings-postgres"]
            P2["postgres<br/>personal data"]
        end
        P1 -.->|system calls| K
        P2 -.->|system calls| K
    end
    style K fill:#f9d5d5,stroke:#c33

Read the diagram backwards: if a web-store process manages to abuse the shared kernel, it is on the same plane as bookings-postgres. The only barrier is how much we have allowed it to ask the kernel for.

  1. The securityContext: pod level and container level

securityContext appears in two places in the manifest, and which fields each one accepts is different. Mixing them up is mistake number one.

apiVersion: v1
kind: Pod
metadata:
  name: example
spec:
  securityContext:            # <-- POD LEVEL: affects every container
    runAsNonRoot: true
    runAsUser: 10001
    fsGroup: 10001
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: app
      image: registry.rutasnorte.example/bookings-api:2.7.1
      securityContext:        # <-- CONTAINER LEVEL: only this container
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop: ["ALL"]

Which field lives at which level

Field Pod Container What it does
runAsUser Yes Yes UID the process runs as
runAsGroup Yes Yes Primary GID
runAsNonRoot Yes Yes The kubelet refuses to start if the UID is 0
supplementalGroups Yes No Additional GIDs
fsGroup Yes No GID applied to the mounted volumes
fsGroupChangePolicy Yes No When the volume permissions are recalculated
seccompProfile Yes Yes System-call filter
seLinuxOptions Yes Yes SELinux labels
sysctls Yes No The pod's kernel parameters
appArmorProfile Yes Yes AppArmor profile (native field since 1.30)
allowPrivilegeEscalation No Yes Sets no_new_privs
capabilities No Yes Add and drop capabilities
privileged No Yes Disables almost all the isolation
readOnlyRootFilesystem No Yes Mounts / read-only
procMount No Yes How /proc is mounted

A mnemonic rule: whatever has to do with identity and with volumes goes on the pod; whatever has to do with what the process can do goes on the container.

Which one wins

When a field exists at both levels and is set in both, the container one wins. This enables a very clean strategy: put the restrictive baseline on the pod and make specific, visible exceptions on the container that needs them.

spec:
  securityContext:
    runAsUser: 10001          # baseline for everyone
  containers:
    - name: app
      # inherits runAsUser: 10001
    - name: log-adapter
      securityContext:
        runAsUser: 10002      # this container overrides it

One confusing nuance: capabilities and allowPrivilegeEscalation are not inherited from the pod because they do not exist at pod level. They must be repeated in every container, including the initContainers and the sidecars. It is tedious and it is the most frequent reason why a "hardened" pod has one unhardened container. Kustomize or Helm help; an admission policy (08-03) guarantees it.

The initContainers count too

spec:
  initContainers:
    - name: prepare-data
      image: registry.rutasnorte.example/utilities:1.4.2
      securityContext:              # it CANNOT be omitted
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop: ["ALL"]
  containers:
    - name: app
      # ...

A privileged initContainer is a privileged pod: it runs with full access before the main container even starts. Go back over 06-04 and check that all your initContainers and sidecars are hardened just like the main containers.

  1. Running without root: runAsNonRoot, runAsUser and runAsGroup

By default a container runs as whatever user the image declares; and a great many public images, nginx included, declare root (UID 0). Running as root inside the container is not the same as being root on the node, but it is far closer than anyone would like:

  • Root inside the container has, by default, a set of capabilities that an ordinary user does not.
  • If the container shares anything with the node (a hostPath, a socket), root writes where an ordinary user could not.
  • When a kernel escape bug appears, most of them require being root inside the container to work. Not being root removes a good part of that class of problem.

The three fields

spec:
  securityContext:
    runAsNonRoot: true      # a check: the kubelet refuses the pod if the UID is 0
    runAsUser: 10001        # the process's effective UID
    runAsGroup: 10001       # primary GID
Field What it does exactly
runAsNonRoot: true It does not change the user. It is a check: if the resulting UID is 0, the kubelet does not start the container
runAsUser: N Forces the UID, ignoring the image's USER
runAsGroup: N Forces the primary GID. If omitted, it usually ends up as 0 (the root group), which is not ideal

runAsNonRoot without runAsUser works only if the image declares a numeric user. If the Dockerfile says USER appuser (a name, not a number), the kubelet cannot resolve the name —it has no access to the image's /etc/passwd before starting it— and the pod fails:

Error: container has runAsNonRoot and image has non-numeric user (appuser),
cannot verify user is non-root

Hence a rule we will pick up again in 08-05: always declare the numeric user in the Dockerfile (USER 10001), and specify runAsUser in the manifest as well. Belt and braces.

If you try to run as root with the check enabled:

Error: container's runAsUser breaks non-root policy

The nginx case in web-store

web-store uses the official nginx image, which starts as root so that it can listen on port 80 and then drops privileges in its worker processes. If we simply add runAsNonRoot: true, it fails when trying to write to /var/cache/nginx and when opening port 80.

There are two roads, and only one is good:

Option What it implies
Add the NET_BIND_SERVICE capability for port 80 It works, but it keeps an unnecessary capability
Use the nginxinc/nginx-unprivileged image and listen on 8080 No capabilities, no root, and the Service translates the port

The second is clearly superior and it is the one we adopt:

# k8s/base/web-store/deployment.yaml (fragment)
spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 101          # the nginx user of the unprivileged image
        runAsGroup: 101
        fsGroup: 101
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: nginx
          image: registry.rutasnorte.example/nginx-unprivileged:1.27.1
          ports:
            - containerPort: 8080     # high port: no capability required

And the Service still publishes port 80 to the outside world:

apiVersion: v1
kind: Service
metadata:
  name: web-store
  namespace: rutas-norte-pro
spec:
  selector:
    app: web-store
  ports:
    - port: 80            # what clients see
      targetPort: 8080    # what the unprivileged container listens on

Nobody outside the cluster notices the difference, and we have removed one capability and the root start-up.

  1. Volumes and file permissions: fsGroup and fsGroupChangePolicy

Here comes the practical problem that makes many people give up and go back to root. It is exactly the case of bookings-postgres.

The problem

A dynamically provisioned volume (module 5) is mounted with whatever permissions the file system gives it, normally owned by root:root with mode 0755. If the container runs as UID 10001, it cannot write to its own data volume.

initdb: error: could not create directory "/var/lib/postgresql/data/pgdata":
Permission denied

The solution: fsGroup

spec:
  securityContext:
    runAsUser: 999
    runAsGroup: 999
    fsGroup: 999        # <-- the key

When fsGroup is present, the kubelet, before starting the containers:

  1. Changes the owning group of the volume's files and directories to that GID.
  2. Adds the write bit for the group.
  3. Sets the setgid bit on the directories, so that new content inherits the group.
  4. Adds that GID to the process's supplementary groups.

The result: the process can write to the volume without being root and without touching the image.

fsGroup does not apply to every volume type. It works with those that have their own file system (most block volumes through CSI, emptyDir, configMap, secret) and not with NFS or other network file systems, where permissions are governed by the server. With NFS you have to coordinate UIDs/GIDs with whoever administers the storage.

fsGroupChangePolicy: the detail that matters in production

Changing the owner of every file is a recursive operation. On the bookings-postgres volume, with millions of files, that can take minutes on every pod start. And it happens on every restart, on every update, on every move between nodes.

spec:
  securityContext:
    fsGroup: 999
    fsGroupChangePolicy: OnRootMismatch    # <-- essential on large volumes
Value Behaviour When to use it
Always (the default) Walks the whole volume on every start Small volumes
OnRootMismatch Checks only the permission of the volume's root directory; if it already matches, it does nothing Large volumes: nearly always the right one

OnRootMismatch works because, if the root directory already has the expected group and permissions, the change must have been made on a previous start. The first time it does the full walk; the following ones are instantaneous.

The complete bookings-postgres case

The postgres:16.4 image defines the postgres user with UID and GID 999. The final configuration:

# k8s/base/bookings-postgres/statefulset.yaml (fragment)
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: bookings-postgres
  namespace: rutas-norte-pro
spec:
  serviceName: bookings-postgres
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 999                      # the image's postgres user
        runAsGroup: 999
        fsGroup: 999                        # the volume becomes writable
        fsGroupChangePolicy: OnRootMismatch  # without this, start-ups of several minutes
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: postgres
          image: registry.rutasnorte.example/postgres:16.4
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: false   # PostgreSQL writes to /var/run and /tmp
            capabilities:
              drop: ["ALL"]
          env:
            # PGDATA in a subdirectory: avoids problems with lost+found
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
        # Metrics exporter sidecar from module 7: hardened as well
        - name: metrics-exporter
          image: registry.rutasnorte.example/postgres-exporter:0.15.0
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 65534
            capabilities:
              drop: ["ALL"]
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: rutasnorte-ssd
        resources:
          requests:
            storage: 50Gi

Four decisions worth justifying:

  1. runAsUser: 999: the UID the image already expects. Using another one would force us to modify the image.
  2. fsGroupChangePolicy: OnRootMismatch: without it, with 50 GiB of personal data, every pod restart would take minutes and would look like a service outage.
  3. readOnlyRootFilesystem: false on PostgreSQL: this is a conscious, documented exception. PostgreSQL writes socket and temporary files outside the data volume. It could be fixed with an emptyDir on /var/run/postgresql and /tmp, and that is the right move in the medium term; here we leave it explicit so you can see that a documented exception is better than a silent one.
  4. The exporter sidecar does carry readOnlyRootFilesystem: true: it writes nothing, so there is no reason to leave it open. Each container is hardened according to what it needs, not according to what its neighbour needs.

  1. allowPrivilegeEscalation and the no_new_privs bit

securityContext:
  allowPrivilegeEscalation: false

This line, which costs nothing, translates in the kernel to the process's no_new_privs bit (documented in prctl(2)). Its exact meaning:

With no_new_privs set, no child process can gain more privileges than its parent, not even by executing a binary with the setuid bit set or with file capabilities.

What a setuid binary is

In Linux, an executable can carry the setuid bit, which makes it run with the UID of its owner rather than that of whoever launched it. The classic example is /usr/bin/passwd: an ordinary user launches it, but it runs as root because it needs to write to /etc/shadow.

Inside a container, a root-owned setuid binary is a bridge between "I am user 10001" and "I am root in this container". Many base images ship several without anyone noticing:

kubectl exec -n rutas-norte-pro deploy/web-store -- \
  find / -xdev -perm -4000 -type f 2>/dev/null
/usr/bin/passwd
/usr/bin/chsh
/usr/bin/gpasswd
/usr/bin/newgrp
/usr/bin/su
/bin/mount
/bin/umount

With allowPrivilegeEscalation: false, running any of them does not raise privileges: the setuid bit is neutralised.

When it should be true

Practically never in an application. There are three legitimate situations:

Situation Why
The container is privileged: true It would be contradictory; Kubernetes forces it to be true
The container adds capabilities with CAP_SYS_ADMIN It usually needs the escalation
Tools that depend on setuid Extremely rare in modern workloads

A consistency note: allowPrivilegeEscalation: false is incompatible with privileged: true. If you set both, the pod is rejected. It is a deliberate check by Kubernetes.

A useful side effect

no_new_privs is also the condition for seccomp to work without special capabilities. Setting it to false prepares the ground for section 9.

Practical rule: allowPrivilegeEscalation: false in every container, with no exceptions that are not documented and approved.

  1. privileged: true and why it is the same as handing over the node

securityContext:
  privileged: true     # almost never the right answer

When a container is privileged:

What gets disabled Consequence
The capability trimming The process has every Linux capability
The device restrictions It sees the node's entire /dev, raw disks included
The default seccomp profile It can make any system call
AppArmor / SELinux No mandatory confinement
Mount restrictions It can mount the node's file systems

The practical consequence is easy to state:

A privileged container is, to all practical effects, root on the node. It can read and write the disks, load kernel modules, reach the host file system and, with it, the mounted secrets of every other pod on that node.

Applied to Rutas Norte: if web-store ran privileged and sat on the same node as bookings-postgres, whoever controlled web-store would have access to the personal data volume. And neither RBAC nor the NetworkPolicies of module 4 would prevent it, because it is not an API request nor network traffic: it is direct disk access.

The legitimate uses and how to reduce them

privileged: true has real uses, all of them at the infrastructure level, not in applications:

Component Why it needs it Finer-grained alternative
CNI plugin (Calico, Cilium) Configures the node's networking NET_ADMIN + hostNetwork
CSI drivers Mount volumes on the node SYS_ADMIN + bidirectional mount
Security agents (Falco) Observe system calls SYS_PTRACE, BPF, PERFMON
Log collector (module 7) Reads the node's /var/log Read-only hostPath, unprivileged

That last one matters and we take advantage of it: the module 7 log collector does not need to be privileged. Mounting /var/log read-only is enough.

# k8s/base/logs/daemonset.yaml (fragment) — hardened version
containers:
  - name: collector
    image: registry.rutasnorte.example/fluent-bit:3.1.7
    securityContext:
      privileged: false                 # NOT needed
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      runAsUser: 0                      # it does need root to read the node's logs
      capabilities:
        drop: ["ALL"]
        add: ["DAC_READ_SEARCH"]        # only to bypass READ permissions
    volumeMounts:
      - name: varlog
        mountPath: /var/log
        readOnly: true                  # <-- read-only
volumes:
  - name: varlog
    hostPath:
      path: /var/log
      type: Directory

It is still a pod with more privileges than an ordinary application —we warned about that in 06-02— but we have come down from "full control of the node" to "reading one directory". That reduction is exactly the work of this lesson.

Rule: if a third-party manifest asks for privileged: true, demand a concrete justification before applying it. Very often it is a convenient default, not a necessity.

  1. Linux capabilities: drop everything and add only what is needed

Historically Linux had two classes of process: root (could do everything) and the rest (could do almost nothing). Capabilities split root's powers into about forty independent pieces that are granted separately.

The relevant capabilities:

Capability What it allows Risk
NET_BIND_SERVICE Listen on ports < 1024 Low
CHOWN Change file ownership Low/medium
DAC_OVERRIDE Bypass all file permissions High
DAC_READ_SEARCH Bypass read permissions Medium
SETUID / SETGID Change identity Medium
NET_RAW Raw sockets (ping, network scanning) Medium
NET_ADMIN Configure networking: interfaces, firewall, routes Very high
SYS_ADMIN A catch-all: mounting, namespaces... Critical
SYS_PTRACE Inspect other processes' memory High
SYS_MODULE Load kernel modules Critical: that is the node
SYS_TIME Change the system clock Medium
BPF, PERFMON eBPF programs and profiling High

A container with no special configuration does not start with all of them: the runtime grants a default set of about fourteen, among them CHOWN, DAC_OVERRIDE, NET_RAW, SETUID, SETGID and NET_BIND_SERVICE. No ordinary web application uses a single one.

The correct practice

securityContext:
  capabilities:
    drop: ["ALL"]      # start from zero
    # add: [...]       # and add only what is strictly indispensable

drop: ["ALL"] followed by nothing is the target configuration for every Rutas Norte component. None of them needs a single capability.

An important detail: drop: ["ALL"] and add: ["NET_BIND_SERVICE"] together do work and are the correct way to express "only this one". The drop is processed before the add.

The NET_BIND_SERVICE example and its better alternative

Suppose you want to keep the official nginx image listening on 80, without being root:

# It works, but it is not the best option
securityContext:
  runAsNonRoot: true
  runAsUser: 101
  allowPrivilegeEscalation: false
  capabilities:
    drop: ["ALL"]
    add: ["NET_BIND_SERVICE"]     # the only way to open port 80 without being root

Let us compare it with the alternative we already adopted:

With NET_BIND_SERVICE on 80 With port 8080
Capabilities One None
Meets PSS restricted (08-03) Yes (it is the only permitted exception) Yes
Changes to the image None nginx configuration or the unprivileged image
What the client sees Port 80 Port 80 (the Service translates it)

The low port is a restriction from an era when servers were shared physical machines. In Kubernetes, where the Service translates ports at no cost, keeping it only adds an unnecessary capability. The general rule: if you can avoid a capability by changing a port number, change it.

Seeing the effective capabilities

kubectl exec -n rutas-norte-pro deploy/bookings-api -- grep Cap /proc/1/status
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000000000000000
CapAmb: 0000000000000000

All zeros: exactly what we want. To translate a non-zero value into names:

kubectl exec -n rutas-norte-pro deploy/bookings-api -- \
  sh -c 'capsh --decode=$(grep CapEff /proc/1/status | cut -f2)'
0x0000000000000000=

If you saw something like cap_chown,cap_dac_override,cap_net_raw,cap_setgid,cap_setuid, then the drop: ["ALL"] is missing from that container.

  1. readOnlyRootFilesystem and how to make it workable

securityContext:
  readOnlyRootFilesystem: true

The container's root file system is mounted read-only. Explicitly mounted volumes remain writable.

Why it is so valuable

Without readOnlyRootFilesystem With readOnlyRootFilesystem: true
A binary can be written into /usr/bin No
A loaded library can be modified No
A configuration file can be overwritten No
A change persists for the container's lifetime Any change is impossible

It turns the container into something immutable at runtime: what is deployed is what runs, from beginning to end. And it gives a very useful property for module 7: if something tries to write to /, an error is logged. That is a free detection signal, and in 08-06 we will turn it into a Falco rule.

The problem and its solution

Almost every piece of software writes something: temporary files, sockets, caches, PIDs. The solution is to mount an emptyDir exactly in those places.

web-store with nginx is the canonical example, because nginx needs four cache directories plus /tmp and its PID file:

# k8s/base/web-store/deployment.yaml (complete fragment)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-store
  namespace: rutas-norte-pro
  labels:
    app: web-store
    app.kubernetes.io/part-of: rutas-norte
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web-store
  template:
    metadata:
      labels:
        app: web-store
        app.kubernetes.io/part-of: rutas-norte
    spec:
      automountServiceAccountToken: false     # from 03-06: it does not talk to the API
      securityContext:
        runAsNonRoot: true
        runAsUser: 101
        runAsGroup: 101
        fsGroup: 101
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: nginx
          image: registry.rutasnorte.example/nginx-unprivileged:1.27.1
          ports:
            - containerPort: 8080
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true      # <-- the goal
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            # Everything nginx needs to write, in memory and ephemeral
            - name: nginx-cache
              mountPath: /var/cache/nginx
            - name: nginx-run
              mountPath: /var/run              # the PID file goes here
            - name: temp
              mountPath: /tmp
          resources:
            requests: { cpu: 50m, memory: 64Mi }
            limits:   { cpu: 500m, memory: 128Mi }
      volumes:
        - name: nginx-cache
          emptyDir:
            medium: Memory                     # in RAM: faster and it never touches the disk
            sizeLimit: 64Mi                    # a limit: an unbounded emptyDir can fill the node
        - name: nginx-run
          emptyDir:
            medium: Memory
            sizeLimit: 8Mi
        - name: temp
          emptyDir:
            sizeLimit: 128Mi

Points worth highlighting:

  • medium: Memory makes the emptyDir live in RAM (tmpfs). For small caches it is faster and avoids writing data to the node's disk, which matters if session information passes through them.
  • sizeLimit is mandatory in practice. An unbounded emptyDir can grow until it exhausts the node's disk (or memory). With medium: Memory it also counts against the pod's memory limit.
  • The emptyDir volumes are emptied when the container restarts, which is exactly what we want: nothing persists.

How to find out which directories are needed

Do not guess: measure. Deploy with readOnlyRootFilesystem: true in rutas-norte-dev and see what fails.

kubectl logs -n rutas-norte-dev deploy/web-store
nginx: [emerg] mkdir() "/var/cache/nginx/client_temp" failed (30: Read-only file system)

The log tells you the exact path. You add the emptyDir, repeat, and in two or three iterations you have the full list. It is a ten-minute procedure per component, done once.

Usual directories by application type:

Application Directories it usually needs
nginx /var/cache/nginx, /var/run, /tmp
Node.js (bookings-api) /tmp (and /home/node/.npm if it installs at runtime, which it should not)
Java /tmp (the JVM writes its performance files there)
Python /tmp; avoid .pyc with PYTHONDONTWRITEBYTECODE=1
PostgreSQL /var/run/postgresql, /tmp, in addition to the data volume
Redis /data (a real volume, not an emptyDir)

  1. Seccomp, AppArmor and SELinux

Seccomp: filtering the system calls

Secure Computing Mode limits which system calls the process can make. It is the most direct defence against kernel bugs: if a bug is in a call your container cannot make, it does not affect you.

spec:
  securityContext:
    seccompProfile:
      type: RuntimeDefault
type value What it does
Unconfined No filter. Every call available
RuntimeDefault The runtime's profile: blocks around 60 dangerous calls
Localhost A custom profile in a file on the node (requires localhostProfile)

Watch out for a historical detail: Kubernetes' default value was Unconfined for years. Since 1.27 the kubelet's SeccompDefault gate allows changing it to RuntimeDefault, but it must be enabled explicitly in the kubelet configuration. Until you do, if you do not set seccompProfile, your container has no seccomp filter at all. This is one of those things people take for granted and should not.

RuntimeDefault blocks calls such as mount, reboot, init_module, kexec_load, ptrace in some contexts, bpf and pivot_root. It is compatible with practically any ordinary application, so setting it is almost free and should always be done.

Custom profiles

When RuntimeDefault is not enough —for instance, for a highly exposed component such as bookings-api— you can write your own profile. The file goes into /var/lib/kubelet/seccomp/profiles/ on each node (usually via a DaemonSet or the Security Profiles Operator):

{
  "defaultAction": "SCMP_ACT_ERRNO",
  "defaultErrnoRet": 1,
  "architectures": ["SCMP_ARCH_X86_64", "SCMP_ARCH_X86", "SCMP_ARCH_X32"],
  "syscalls": [
    {
      "names": [
        "accept4", "bind", "listen", "socket", "connect", "getsockname",
        "read", "write", "readv", "writev", "close", "openat", "fstat",
        "epoll_create1", "epoll_ctl", "epoll_pwait", "futex",
        "mmap", "munmap", "mprotect", "brk", "rt_sigaction", "rt_sigprocmask",
        "clock_gettime", "getpid", "gettid", "exit", "exit_group",
        "nanosleep", "sched_yield", "madvise", "getrandom", "fcntl"
      ],
      "action": "SCMP_ACT_ALLOW"
    }
  ]
}
spec:
  securityContext:
    seccompProfile:
      type: Localhost
      localhostProfile: profiles/bookings-api.json   # path relative to the seccomp directory

Serious warnings about custom profiles:

  • They are fragile. An update to the standard library or the runtime can introduce a new call and the process dies with SIGSYS or an incomprehensible EPERM.
  • They have to be generated by observation, not written by hand. The Security Profiles Operator can record the profile of a running workload.
  • Always start with RuntimeDefault. Moving to a custom profile only pays off on very critical components and where there is capacity to maintain it.

For debugging you can use defaultAction: SCMP_ACT_LOG, which logs instead of blocking. Never leave that in production: it protects against nothing.

AppArmor

Path-based mandatory access control, common on Debian and Ubuntu. Since Kubernetes 1.30 it has a native field in the securityContext, with no need for annotations:

spec:
  securityContext:
    appArmorProfile:
      type: RuntimeDefault
  containers:
    - name: app
      securityContext:
        appArmorProfile:
          type: Localhost
          localhostProfile: rutasnorte-api      # profile loaded on the node

The container.apparmor.security.beta.kubernetes.io/<container> annotations still work for compatibility, but they are deprecated: use the field.

SELinux

Label-based mandatory access control, common on Red Hat and derivatives. It is configured with seLinuxOptions:

spec:
  securityContext:
    seLinuxOptions:
      level: "s0:c123,c456"

In practice, the most frequent use in Kubernetes is to let the runtime assign labels automatically. Changing them by hand is usually counterproductive unless the organisation has its own SELinux policy.

Comparison

Seccomp AppArmor SELinux
What it controls System calls File paths and capabilities Labels on every object
Where it is common Every distribution Debian, Ubuntu, SUSE RHEL, Fedora, CentOS
Difficulty Low with RuntimeDefault Medium High
Portability between clusters High Depends on the node Depends on the node
Recommendation RuntimeDefault always If your nodes ship it If your organisation already uses it

Recommendation for Rutas Norte: seccompProfile: RuntimeDefault on absolutely every pod, and AppArmor/SELinux according to what the nodes provide, without relying on them for the essential security (because the local cluster is minikube and the production one might not match).

  1. The pod settings to avoid

There are five settings that break the pod's isolation from the node. We have mentioned them along the way in 05-01 and 06-02; here we gather them together.

Setting What it breaks Concrete risk in Rutas Norte
hostNetwork: true The network namespace The pod sees all the node's traffic and bypasses the NetworkPolicies of 04-06
hostPID: true The PID namespace It sees the other pods' processes, with their command lines and their memory
hostIPC: true The IPC namespace It reaches the shared memory of other processes on the node
hostPath The mount namespace Reads or writes the node's disk, including other pods' mounted secrets
hostPort The port allocation Opens a port on the node's IP, bypassing the Service and the Ingress

hostNetwork and its interaction with NetworkPolicies

This one deserves its own paragraph because it dismantles the work of an earlier module. In 04-06 we built a deny-all policy in rutas-norte-pro and then authorised each conversation one at a time. A pod with hostNetwork: true uses the node's network stack, not the pod's, so NetworkPolicies —which act on pod IPs— do not apply to it. All that work is voided for that pod.

It is the reason why a DaemonSet with hostNetwork must not share a namespace with the applications, and why its manifests deserve especially careful review.

hostPath: the warning from 05-01, expanded

# NEVER in production for an application
volumes:
  - name: dangerous
    hostPath:
      path: /            # the node's entire disk
      type: Directory

A hostPath to / gives access to:

  • /var/lib/kubelet/pods/*/volumes/kubernetes.io~projected/the ServiceAccount tokens of every pod on the node.
  • /etc/kubernetes/ — on a control plane node, the cluster's keys.
  • /var/lib/docker or /var/lib/containerd — every image and layer.
  • /var/run/containerd/containerd.sock — the runtime socket, which allows starting privileged containers.

That last point is key and is often overlooked: mounting the runtime socket is equivalent to privileged: true, even if the pod does not declare it. Any manifest that mounts /var/run/docker.sock or containerd.sock must be treated as a privileged pod.

The legitimate uses of hostPath are infrastructure ones (reading /var/log for the collector, exposing devices to a CSI driver) and they must always be:

  1. The most specific path possible, never / nor /var.
  2. With readOnly: true whenever possible.
  3. With an explicit type: (Directory, File, Socket) so that it is not created by accident.
  4. In a separate namespace with a relaxed security profile (08-03).

hostPort

ports:
  - containerPort: 8080
    hostPort: 8080      # avoid it

It opens the port on the node's IP. The problems: only one pod per node fits with that port, it bypasses the Service and the Ingress (and therefore the TLS of 04-05 and the WAF of 08-04), and it exposes the service to anyone who can reach the node's IP. To publish services you use Ingress, as in module 4.

  1. RuntimeClass and sandboxed runtimes

Everything above hardens a container, but it does not change the fundamental fact: the kernel is shared. If you need strong isolation, something else is required.

A RuntimeClass selects which runtime the pod uses:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc          # configured in containerd on the node
apiVersion: v1
kind: Pod
metadata:
  name: isolated-workload
spec:
  runtimeClassName: gvisor      # this pod uses gVisor instead of runc
  containers:
    - name: app
      image: registry.rutasnorte.example/third-party-processor:1.2.0

The options

Runtime How it isolates Start-up cost I/O cost Compatibility
runc (the default) Namespaces + cgroups Minimal Minimal Total
gVisor (runsc) A user-space kernel that intercepts the calls +50-100 ms Noticeable on I/O-heavy work High, with exceptions
Kata Containers A lightweight virtual machine per pod +200-500 ms Low with virtio Very high
Firecracker Micro-VM (the basis of Kata and of several cloud services) +125 ms Low High

When it pays off

Workload Reinforced isolation?
web-store, bookings-api, notifications-worker No. Our own reviewed code; ordinary hardening is enough
bookings-postgres No. The I/O cost would be unacceptable; it is protected with RBAC, networking and encryption
Running code submitted by third parties Yes, essential
Multi-tenancy with customers who do not know each other Yes
Parsing a customer file with a complex library Probably yes

Rutas Norte has no workload today that justifies it. If tomorrow it let the bus companies upload a timetable file and processed it with a complex parsing library, that component would be the natural candidate: it runs code over data that comes from outside.

The important thing is to know the tool and its criterion: reinforced isolation is for code you do not trust, not for your own.

  1. The full hardening of Rutas Norte

We bring everything together in the final configuration of each component, with the justification for every exception.

Summary table

Component UID RootFS RO Caps Seccomp Exceptions
web-store 101 Yes drop: ALL RuntimeDefault 3 emptyDir
bookings-api 10001 Yes drop: ALL RuntimeDefault emptyDir on /tmp
bookings-postgres 999 No drop: ALL RuntimeDefault Writable RootFS (documented)
redis-cache 999 Yes drop: ALL RuntimeDefault Volume on /data
notifications-worker 10002 Yes drop: ALL RuntimeDefault emptyDir on /tmp
occupancy-reports 10003 Yes drop: ALL RuntimeDefault emptyDir for the report
Payments ambassador 10004 Yes drop: ALL RuntimeDefault None
Log collector 0 Yes drop: ALL + DAC_READ_SEARCH RuntimeDefault Read-only hostPath

bookings-api: the complete case

# k8s/base/bookings-api/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bookings-api
  namespace: rutas-norte-pro
  labels:
    app: bookings-api
    app.kubernetes.io/part-of: rutas-norte
    environment: pro
spec:
  replicas: 4
  selector:
    matchLabels:
      app: bookings-api
  template:
    metadata:
      labels:
        app: bookings-api
        app.kubernetes.io/part-of: rutas-norte
        environment: pro
    spec:
      serviceAccountName: bookings-api
      automountServiceAccountToken: true    # it does read a ConfigMap through the API (see 08-01)
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
        fsGroup: 10001
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: api
          image: registry.rutasnorte.example/bookings-api@sha256:9f2c1d4e8a7b6035c1e4d9a2b8f7c3e6d5a4b3c2e1f0a9b8c7d6e5f4a3b2c1d0
          ports:
            - name: http
              containerPort: 8080
            - name: metrics
              containerPort: 9090
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          # Probes from module 7, unchanged
          livenessProbe:
            httpGet: { path: /health, port: http }
            initialDelaySeconds: 10
            periodSeconds: 10
          readinessProbe:
            httpGet: { path: /ready, port: http }
            periodSeconds: 5
          volumeMounts:
            - name: temp
              mountPath: /tmp
          resources:
            requests: { cpu: 200m, memory: 256Mi }
            limits:   { cpu: "1",  memory: 512Mi }
        # Ambassador container towards the external payment gateway (06-04)
        - name: payments-ambassador
          image: registry.rutasnorte.example/payments-ambassador:1.3.0
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 10004
            capabilities:
              drop: ["ALL"]
          resources:
            requests: { cpu: 20m, memory: 32Mi }
            limits:   { cpu: 100m, memory: 64Mi }
      volumes:
        - name: temp
          emptyDir:
            sizeLimit: 64Mi

Note the image referenced by digest instead of by tag. That is the subject of 08-05 and it fits here naturally: there is no point in hardening a container if you do not know for certain which image is running.

notifications-worker with its log adapter

# k8s/base/notifications-worker/deployment.yaml (fragment)
spec:
  template:
    spec:
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 10002
        runAsGroup: 10002
        fsGroup: 10002
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: worker
          image: registry.rutasnorte.example/notifications-worker:3.2.0
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: { drop: ["ALL"] }
          volumeMounts:
            - name: temp
              mountPath: /tmp
            - name: worker-logs
              mountPath: /var/log/worker
        # Adapter that converts the logs to JSON (06-04 and 07-05)
        - name: log-adapter
          image: registry.rutasnorte.example/log-adapter:1.1.0
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 10002        # same UID: it shares the log volume
            capabilities: { drop: ["ALL"] }
          volumeMounts:
            - name: worker-logs
              mountPath: /var/log/worker
              readOnly: true        # it only reads: there is no reason to write
      volumes:
        - name: temp
          emptyDir: { sizeLimit: 64Mi }
        - name: worker-logs
          emptyDir: { sizeLimit: 256Mi }

Two fine details: the adapter uses the same UID as the worker so that it can read the files the worker writes (alternative: a shared fsGroup), and it mounts the volume with readOnly: true because it only reads. Hardening a sidecar is just as important as hardening the main container: they share the network namespace and often volumes too.

The occupancy-reports CronJob

# k8s/base/occupancy-reports/cronjob.yaml (fragment)
apiVersion: batch/v1
kind: CronJob
metadata:
  name: occupancy-reports
  namespace: rutas-norte-pro
spec:
  schedule: "0 3 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          serviceAccountName: occupancy-reports
          automountServiceAccountToken: false     # it does not talk to the API (08-01)
          securityContext:
            runAsNonRoot: true
            runAsUser: 10003
            runAsGroup: 10003
            fsGroup: 10003
            seccompProfile:
              type: RuntimeDefault
          containers:
            - name: reports
              image: registry.rutasnorte.example/occupancy-reports:1.5.0
              securityContext:
                allowPrivilegeEscalation: false
                readOnlyRootFilesystem: true
                capabilities: { drop: ["ALL"] }
              volumeMounts:
                - name: work
                  mountPath: /work
          volumes:
            - name: work
              emptyDir: { sizeLimit: 512Mi }

This component is particularly sensitive because it reads personal data in order to aggregate it. Remember what we said in 07-05: the reports contain only aggregates (occupancy per route and per time slot), never individual records, and none of it goes into the logs.

A reusable fragment

Since these blocks repeat, the sensible thing is to extract them. With Kustomize (module 10) a strategic patch is applied to every Deployment:

# k8s/base/common-hardening.yaml
# Reference block. Copy it or apply it as a patch.
podSecurityContext: &pod
  runAsNonRoot: true
  seccompProfile:
    type: RuntimeDefault

containerSecurityContext: &container
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop: ["ALL"]

And in 08-03 we will take the definitive step: making the cluster reject any pod that does not meet these rules, so that no future deployment can slip past them by oversight.

  1. How to check the result from inside the container

Unverified hardening is an assumption. These checks take a minute and should be done after every change.

Who am I

kubectl exec -n rutas-norte-pro deploy/bookings-api -c api -- id
uid=10001 gid=10001 groups=10001

If you saw uid=0(root), then runAsUser is not being applied (check that the container's securityContext is not overriding it).

The root file system is read-only

kubectl exec -n rutas-norte-pro deploy/bookings-api -c api -- \
  sh -c 'touch /write-test 2>&1 || echo "correct: blocked"'
touch: /write-test: Read-only file system
correct: blocked

And that the emptyDir volumes really are writable:

kubectl exec -n rutas-norte-pro deploy/bookings-api -c api -- \
  sh -c 'touch /tmp/ok && echo "correct: /tmp writable" && rm /tmp/ok'
correct: /tmp writable

Effective capabilities

kubectl exec -n rutas-norte-pro deploy/bookings-api -c api -- \
  grep -E 'CapEff|CapBnd|NoNewPrivs' /proc/1/status
CapEff:	0000000000000000
CapBnd:	0000000000000000
NoNewPrivs:	1

CapEff: 0 confirms the drop: ["ALL"]. NoNewPrivs: 1 confirms allowPrivilegeEscalation: false. It is the most direct check of the two most important lines in the manifest.

Seccomp is active

kubectl exec -n rutas-norte-pro deploy/bookings-api -c api -- grep Seccomp /proc/1/status
Seccomp:	2
Seccomp_filters:	1
Seccomp value Meaning
0 Disabled: the profile is not being applied
1 Strict mode (rare)
2 Filter mode: RuntimeDefault or Localhost active

If you see 0 when you expected RuntimeDefault, check that the field is at the right level and that the node supports it.

The namespace isolation works

kubectl exec -n rutas-norte-pro deploy/bookings-api -c api -- ps aux
PID   USER     TIME  COMMAND
    1 10001     0:04 node /app/server.js
   28 10001     0:00 ps aux

Only the container's processes. If you saw node processes (kubelet, containerd, processes from other pods), then hostPID: true is enabled.

A complete verification script

#!/usr/bin/env bash
# k8s/security/verify-hardening.sh
# Checks the effective hardening of a running container.
set -uo pipefail

NS="${1:-rutas-norte-pro}"
WORKLOAD="${2:-deploy/bookings-api}"
CONTAINER="${3:-api}"

run_in() { kubectl exec -n "$NS" "$WORKLOAD" -c "$CONTAINER" -- "$@" 2>/dev/null; }

echo "=== $NS / $WORKLOAD / $CONTAINER ==="

echo -n "User:           "; run_in id

echo -n "RootFS RO:      "
if run_in sh -c 'touch /.test' >/dev/null 2>&1; then
  echo "NO (writable) <-- REVIEW"; run_in rm -f /.test
else
  echo "yes"
fi

echo -n "Capabilities:   "
caps=$(run_in grep CapEff /proc/1/status | awk '{print $2}')
[[ "$caps" == "0000000000000000" ]] && echo "none (correct)" || echo "$caps <-- REVIEW"

echo -n "NoNewPrivs:     "
nnp=$(run_in grep NoNewPrivs /proc/1/status | awk '{print $2}')
[[ "$nnp" == "1" ]] && echo "1 (correct)" || echo "$nnp <-- REVIEW"

echo -n "Seccomp:        "
sec=$(run_in grep '^Seccomp:' /proc/1/status | awk '{print $2}')
[[ "$sec" == "2" ]] && echo "filter active (correct)" || echo "$sec <-- REVIEW"

echo -n "Processes seen: "; run_in sh -c 'ps aux | wc -l'
=== rutas-norte-pro / deploy/bookings-api / api ===
User:           uid=10001 gid=10001 groups=10001
RootFS RO:      yes
Capabilities:   none (correct)
NoNewPrivs:     1 (correct)
Seccomp:        filter active (correct)
Processes seen: 3

A practical note: this script needs pods/exec, a permission that in 08-01 we did not give to support or development. It is a platform task, and the sensible thing is to integrate it as an automatic check in rutas-norte-pre before promoting to production.

Common Mistakes and Tips

Putting capabilities or readOnlyRootFilesystem in the pod's securityContext. They do not exist at that level. The manifest is either rejected or the field is ignored, depending on the tool. They go on each container.

Forgetting the initContainers and the sidecars. A pod with a perfectly hardened main container and a privileged initContainer is a privileged pod. Go back over 06-04.

Using runAsNonRoot: true with an image that declares USER by name. The kubelet cannot verify it and the pod does not start. Always use a numeric UID in the Dockerfile (08-05) and runAsUser in the manifest.

Giving up on readOnlyRootFilesystem and removing it. The log tells you the exact path that fails. Two or three emptyDir volumes solve almost every case.

emptyDir without sizeLimit. It can fill the node's disk or memory and cause other pods to be evicted. With medium: Memory it also counts against the pod's memory limit.

Taking it for granted that seccomp is active. The default value was Unconfined for years. If you do not set seccompProfile: RuntimeDefault, there is probably no filter. Check it with grep Seccomp /proc/1/status.

Writing a custom seccomp profile by hand. It is fragile and breaks with every image update. Start with RuntimeDefault; if you need more, generate the profile by observing the workload.

Mounting the runtime socket. /var/run/docker.sock or containerd.sock are equivalent to privileged: true even if the manifest does not say so.

Accepting privileged: true in a third-party manifest without asking. Very often it is convenience, not necessity. Ask for the justification.

Forgetting fsGroupChangePolicy: OnRootMismatch on large volumes. The pod takes minutes to start on every restart and looks like an outage.

Believing that runAsNonRoot changes the user. It does not change it: it verifies it. Without runAsUser or a numeric USER in the image, it does not start.

Assuming a container isolates like a VM. It shares the kernel. That is the premise of the whole lesson and the reason nodes must be kept patched.

Golden tip: the target for every application is exactly this block, and any deviation must be commented in the YAML itself explaining why:

securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  runAsNonRoot: true
  capabilities:
    drop: ["ALL"]
  seccompProfile:
    type: RuntimeDefault

Exercises

Exercise 1: harden redis-cache

redis-cache is a single-replica StatefulSet using the redis:7.4-alpine image, which runs as root and mounts a PVC on /data. Write the complete securityContext fragment (pod and container level) knowing that:

  • The official Redis image defines the redis user with UID and GID 999.
  • Redis writes its data file to /data (the PVC) and its PID file to /var/run/redis.
  • It needs no Linux capability whatsoever.
  • The volume is 8 GiB.

State what would break if you omitted fsGroup and how you would detect it.

Exercise 2: diagnose a pod that will not start

After hardening notifications-worker, the pod ends up in CrashLoopBackOff:

NAME                                     READY   STATUS             RESTARTS   AGE
notifications-worker-7d9c8b6f5-x2k4m     0/2     CrashLoopBackOff   4          2m
kubectl logs -n rutas-norte-pro notifications-worker-7d9c8b6f5-x2k4m -c worker
Error: EACCES: permission denied, open '/app/queue/pending.dat'
    at Object.openSync (node:fs:596:3)
    at savePending (/app/lib/queue.js:44:18)

The applied securityContext is:

spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 10002
    seccompProfile: { type: RuntimeDefault }
  containers:
    - name: worker
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities: { drop: ["ALL"] }
  1. What is the exact cause?
  2. Propose two solutions and argue which is better.
  3. Write the command that would confirm the diagnosis.

Exercise 3: audit the unhardened pods in the cluster

Write a command that lists every pod in rutas-norte-pro that breaches any of these conditions, stating which one:

  • readOnlyRootFilesystem: true on all its containers.
  • allowPrivilegeEscalation: false on all its containers.
  • capabilities.drop includes ALL on all its containers.
  • It does not use hostNetwork, hostPID or hostIPC.
  • No container is privileged.

It must walk the initContainers as well.

Solutions

Solution 1

# k8s/base/redis-cache/statefulset.yaml (fragment)
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis-cache
  namespace: rutas-norte-pro
spec:
  serviceName: redis-cache
  replicas: 1
  template:
    spec:
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
        runAsGroup: 999
        fsGroup: 999                        # makes the /data PVC writable
        fsGroupChangePolicy: OnRootMismatch # 8 GiB: avoids slow walks
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: redis
          image: registry.rutasnorte.example/redis:7.4-alpine
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - name: data
              mountPath: /data              # PVC: writable through fsGroup
            - name: run
              mountPath: /var/run/redis     # emptyDir: the PID file
            - name: temp
              mountPath: /tmp
          resources:
            requests: { cpu: 100m, memory: 256Mi }
            limits:   { cpu: 500m, memory: 512Mi }
      volumes:
        - name: run
          emptyDir: { medium: Memory, sizeLimit: 8Mi }
        - name: temp
          emptyDir: { sizeLimit: 32Mi }
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: rutasnorte-ssd
        resources:
          requests:
            storage: 8Gi

Without fsGroup: the PVC is mounted owned by root:root with mode 0755. Redis, running as UID 999, cannot write to /data. The pod starts and dies as soon as it tries to persist:

Can't open the append-only file: Permission denied

Detection:

kubectl logs -n rutas-norte-pro redis-cache-0
kubectl exec -n rutas-norte-pro redis-cache-0 -- ls -ld /data
drwxr-xr-x 2 root root 4096 Aug  6 03:11 /data

With fsGroup: 999 applied:

drwxrwsr-x 2 root 999 4096 Aug  6 03:14 /data

The group becomes 999, the group write bit appears and the s indicates the setgid on the directory.

An additional note: since it is a cache, losing the volume is not critical. But it is critical that it does not start, because bookings-api would respond more slowly and seat availability could degrade.

Solution 2

1. The cause. readOnlyRootFilesystem: true prevents writing anywhere in the root file system that is not a mounted volume. The worker tries to write to /app/queue/pending.dat, which is inside the image and therefore on the read-only rootfs. The EACCES message on a write openSync is the typical signature of this problem. It is not a UID problem: even running as root, / would still be read-only.

2. Two solutions:

Option A — mount an emptyDir on /app/queue:

      containers:
        - name: worker
          volumeMounts:
            - name: queue
              mountPath: /app/queue
      volumes:
        - name: queue
          emptyDir: { sizeLimit: 256Mi }

Option B — change the worker's configuration so that it uses /tmp, already mounted:

          env:
            - name: QUEUE_PATH
              value: /tmp/queue

Which is better: it depends on whether that data must survive. And here there is a business consideration that goes beyond the YAML: /app/queue/pending.dat holds notifications waiting to be sent. With either option, that data is lost when the container restarts, because emptyDir is ephemeral. If a customer buys a ticket and the worker restarts before sending the email, that customer never gets their confirmation.

The correct solution, therefore, is C: notifications-worker should not have a local queue at all. The queue must live in a shared, durable store —Redis with persistence, or the database itself— so that any replica can pick the work up. That also fixes a problem that existed before any hardening: with several replicas, each one had its own local queue, invisible to the others.

As an immediate unblocking measure, A with an emptyDir is preferable to B, because it leaves the problem visible in the manifest (volumes: queue) instead of hiding it in an environment variable. And a ticket must be opened for C.

This exercise illustrates something important: hardening a container often uncovers pre-existing design flaws. The readOnlyRootFilesystem did not create the problem; it made it visible.

3. Confirming the diagnosis:

# Start temporarily without readOnlyRootFilesystem in rutas-norte-dev and check
kubectl exec -n rutas-norte-dev deploy/notifications-worker -c worker -- \
  sh -c 'ls -ld /app/queue; touch /app/queue/test && echo WRITABLE || echo BLOCKED'
drwxr-xr-x 2 10002 10002 4096 Aug  6 02:58 /app/queue
BLOCKED

The directory permissions are correct (owner 10002, the same UID as the process), and it still fails: that rules out a permissions problem and confirms that the cause is the read-only mount.

Solution 3

#!/usr/bin/env bash
# k8s/security/audit-hardening.sh
# Lists the pods that breach the hardening baseline.
set -euo pipefail
NS="${1:-rutas-norte-pro}"

kubectl get pods -n "$NS" -o json | jq -r '
  .items[] as $pod
  | ($pod.metadata.name) as $name
  | [
      # Pod-level settings
      (if $pod.spec.hostNetwork == true then "hostNetwork" else empty end),
      (if $pod.spec.hostPID     == true then "hostPID"     else empty end),
      (if $pod.spec.hostIPC     == true then "hostIPC"     else empty end),
      (if ($pod.spec.volumes // [] | map(select(.hostPath)) | length) > 0
         then "hostPath" else empty end),

      # Per-container settings, including the initContainers
      ( (($pod.spec.containers // []) + ($pod.spec.initContainers // []))[]
        | . as $c
        | (
            (if ($c.securityContext.privileged // false) == true
               then "privileged:\($c.name)" else empty end),
            (if ($c.securityContext.readOnlyRootFilesystem // false) != true
               then "rootfs-writable:\($c.name)" else empty end),
            (if ($c.securityContext.allowPrivilegeEscalation // true) != false
               then "escalation-allowed:\($c.name)" else empty end),
            (if (($c.securityContext.capabilities.drop // []) | index("ALL")) == null
               then "no-drop-ALL:\($c.name)" else empty end)
          )
      )
    ] as $problems
  | if ($problems | length) > 0
      then "\($name)\n    " + ($problems | join("\n    "))
      else empty
    end'
bash k8s/security/audit-hardening.sh rutas-norte-pro
bookings-postgres-0
    rootfs-writable:postgres
fluent-bit-collector-nx7q2
    hostPath

How to read this particular output:

  • bookings-postgres: this is the documented exception from section 12. It is justified, although the task of resolving it with an emptyDir is still open.
  • fluent-bit-collector: the logging DaemonSet with a read-only hostPath. Legitimate and already reduced to the minimum (no privileged), but in 08-03 we will move it to a separate namespace so that it does not relax the profile of the whole of rutas-norte-pro.

The fact that no other pod appears is the evidence that the hardening is applied. This script should run in the pipeline against rutas-norte-pre before every promotion to production, with a list of known exceptions so that it only alerts on new ones.

A note on the // true and // false in the jq: the default values are written in the unsafe direction on purpose. An absent allowPrivilegeEscalation means allowed, so its default is true; an absent readOnlyRootFilesystem means writable, so its default is false. An audit must assume the worst when the field is missing.

Conclusion

We have added the second layer of defence. Going over the essentials:

  • A container is not a virtual machine. It shares a kernel with the node, and the isolation comes from namespaces, cgroups and the limits on what it can ask the kernel for. That is why patching the nodes is a top-tier security task.
  • The securityContext lives at two levels: identity and volumes on the pod; capabilities, escalation and file system on the container. The container's wins, and capabilities, privileged, readOnlyRootFilesystem and allowPrivilegeEscalation must be repeated on every container, initContainers and sidecars included.
  • Do not run as root (runAsNonRoot + a numeric runAsUser), and solve the volume permissions with fsGroup and fsGroupChangePolicy: OnRootMismatch when the volume is large, as in bookings-postgres.
  • allowPrivilegeEscalation: false sets no_new_privs and neutralises setuid binaries. It costs one line.
  • privileged: true is the same as handing over the node, and mounting the runtime socket is too, even if it is not declared.
  • capabilities: drop: ["ALL"] and add only what is indispensable; and very often even NET_BIND_SERVICE can be avoided by moving to a high port.
  • readOnlyRootFilesystem: true makes the container immutable at runtime, and it becomes workable with a handful of emptyDir volumes with sizeLimit.
  • seccompProfile: RuntimeDefault always; do not take it for granted that it is already active.
  • Avoid hostNetwork, hostPID, hostIPC, hostPath and hostPort; remember that hostNetwork voids the NetworkPolicies of module 4.
  • For code you do not trust, RuntimeClass with gVisor or Kata; for your own, it does not pay off.
  • And check it all from the inside: id, the failed write, CapEff, NoNewPrivs and Seccomp in /proc/1/status.

Every Rutas Norte component is now hardened. But look at the nature of what we have done: we have written a correct YAML. Nothing stops somebody tomorrow from deploying a new pod without a securityContext, or copying a manifest from the internet with privileged: true, or a colleague adding a sidecar and forgetting the drop: ["ALL"]. All the hardening in this lesson depends on each person remembering, every time. And that is not a guarantee: it is a hope.

The next lesson, 08-03, Pod Security Policies and Pod Security Standards, takes the definitive step: moving from "each team sets its securityContext properly" to "the cluster does not accept a pod that lacks it". We will see the Pod Security Standards, the Pod Security Admission built into the apiserver, and policy engines such as Kyverno that let you enforce any rule you can think of, with the exact rejection message the API returns when a pod does not comply.

Kubernetes Course

Module 1: Introduction to Kubernetes

Module 2: Core Kubernetes Components

Module 3: Configuration and Secret Management

Module 4: Networking in Kubernetes

Module 5: Storage in Kubernetes

Module 6: Advanced Kubernetes Concepts

Module 7: Monitoring and Logging

Module 8: Kubernetes Security

Module 9: Scaling and Performance

Module 10: Kubernetes Ecosystem and Tooling

Module 11: Case Studies and Real-World Applications

Module 12: Preparing for Kubernetes Certification

© Copyright 2026. All rights reserved