In the previous lesson we closed the door on who can do what: RBAC decides whether you have the right to create a pod in rutas-norte-pro. But we finished with a very specific discomfort. Someone from platform, with perfectly legitimate permissions, can deploy a container today with privileged: true, mounting the node's disk with hostPath and running as root. RBAC will say yes, because they have permission to create pods. And from there the whole node —with every pod running on it, bookings-postgres and its personal data included— is laid bare.
This lesson is about the second barrier: what the process can do once it is running inside the container. It is the difference between "somebody has managed to run code in web-store" —a serious but contained incident— and "somebody has managed to run code in web-store and from there read the customer database" —an incident of another magnitude entirely—.
The tool is called securityContext and it is, alongside RBAC, what delivers the most security value per line of YAML written.
Important warning. This lesson explains the hardening mechanisms and proposes an example configuration. The real hardening of a production platform must be reviewed by a security professional, who knows the organisation's specific threat model. When the system handles personal data —as
bookings-postgresdoes— the design must also be known to the compliance officer. The approach here is exclusively defensive: understanding the mechanisms in order to close them, never to exploit them.
Contents
- What a container actually isolates
- The
securityContext: pod level and container level - Running without root:
runAsNonRoot,runAsUserandrunAsGroup - Volumes and file permissions:
fsGroupandfsGroupChangePolicy allowPrivilegeEscalationand theno_new_privsbitprivileged: trueand why it is the same as handing over the node- Linux capabilities: drop everything and add only what is needed
readOnlyRootFilesystemand how to make it workable- Seccomp, AppArmor and SELinux
- The pod settings to avoid
RuntimeClassand sandboxed runtimes- The full hardening of Rutas Norte
- How to check the result from inside the container
- Common mistakes and tips
- Exercises
- Conclusion
- What a container actually isolates
Before configuring anything you have to understand what you are configuring. And the starting statement is uncomfortable:
A container is not a virtual machine. All the containers on a node share the same Linux kernel. A container is an ordinary process of the node's operating system, restricted in three ways: what it sees, what it consumes and what it can ask the kernel for.
Compare it with a virtual machine:
| Virtual machine | Container | |
|---|---|---|
| Kernel | Its own, isolated | Shared with the node |
| Isolation boundary | Hypervisor (hardware) | Kernel features (software) |
| Attack surface | The hypervisor interface (small) | Every system call (~350) |
| Start-up | Seconds or minutes | Milliseconds |
| Footprint | A complete operating system | Just the process |
| Escape | Very difficult | Possible with a kernel bug or a misconfiguration |
The three mechanisms that provide the isolation:
Kernel namespaces: what the process sees
Not to be confused with Kubernetes Namespace objects: they are a completely different Linux concept. A kernel namespace gives the process its own view of a system resource.
| Namespace | What it isolates | The Kubernetes setting that breaks it |
|---|---|---|
| PID | The visible processes | hostPID: true |
| Network | Interfaces, IP, ports, routing tables | hostNetwork: true |
| Mount | The file tree | hostPath (partially) |
| IPC | Shared memory, message queues | hostIPC: true |
| UTS | Host name | — |
| User | UID mapping | (user namespaces, still optional) |
When you run ps aux inside a container and see only your process as PID 1, that is the PID namespace at work. When hostPID: true is enabled, you see every process on the node, including those of the other pods, with their command lines and their environment variables.
cgroups: how much it can consume
Control groups limit CPU, memory, I/O and the number of processes. They are what sits behind the resources.limits of module 3. Their purpose is mainly stability —stopping one pod from taking the node down— but also security, because a local denial of service is an attack.
Capabilities and system-call filters: what it can ask the kernel for
This is where the securityContext makes the difference and where the game is won or lost. We will come back to it in sections 7 and 9.
What this means in practice
The fact that all containers share a kernel has a direct consequence:
- A Linux kernel bug can allow an escape from the container. Such bugs turn up periodically and get fixed; that is why keeping the nodes patched is a first-order security task, not routine maintenance.
- The less a container can ask the kernel for, the less surface it has to take advantage of one of those bugs. That is the logic behind everything that follows.
- If you need strong isolation for something you do not control, ordinary containers are not the answer: you need a sandboxed runtime (section 11) or an outright virtual machine.
flowchart TB
subgraph Node["Kubernetes node"]
K["Linux kernel — SHARED"]
subgraph C1["Pod web-store"]
P1["nginx<br/>PID, net, mount ns<br/>cgroups<br/>reduced capabilities<br/>seccomp"]
end
subgraph C2["Pod bookings-postgres"]
P2["postgres<br/>personal data"]
end
P1 -.->|system calls| K
P2 -.->|system calls| K
end
style K fill:#f9d5d5,stroke:#c33
Read the diagram backwards: if a web-store process manages to abuse the shared kernel, it is on the same plane as bookings-postgres. The only barrier is how much we have allowed it to ask the kernel for.
- The
securityContext: pod level and container level
securityContext: pod level and container levelsecurityContext appears in two places in the manifest, and which fields each one accepts is different. Mixing them up is mistake number one.
apiVersion: v1
kind: Pod
metadata:
name: example
spec:
securityContext: # <-- POD LEVEL: affects every container
runAsNonRoot: true
runAsUser: 10001
fsGroup: 10001
seccompProfile:
type: RuntimeDefault
containers:
- name: app
image: registry.rutasnorte.example/bookings-api:2.7.1
securityContext: # <-- CONTAINER LEVEL: only this container
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]Which field lives at which level
| Field | Pod | Container | What it does |
|---|---|---|---|
runAsUser |
Yes | Yes | UID the process runs as |
runAsGroup |
Yes | Yes | Primary GID |
runAsNonRoot |
Yes | Yes | The kubelet refuses to start if the UID is 0 |
supplementalGroups |
Yes | No | Additional GIDs |
fsGroup |
Yes | No | GID applied to the mounted volumes |
fsGroupChangePolicy |
Yes | No | When the volume permissions are recalculated |
seccompProfile |
Yes | Yes | System-call filter |
seLinuxOptions |
Yes | Yes | SELinux labels |
sysctls |
Yes | No | The pod's kernel parameters |
appArmorProfile |
Yes | Yes | AppArmor profile (native field since 1.30) |
allowPrivilegeEscalation |
No | Yes | Sets no_new_privs |
capabilities |
No | Yes | Add and drop capabilities |
privileged |
No | Yes | Disables almost all the isolation |
readOnlyRootFilesystem |
No | Yes | Mounts / read-only |
procMount |
No | Yes | How /proc is mounted |
A mnemonic rule: whatever has to do with identity and with volumes goes on the pod; whatever has to do with what the process can do goes on the container.
Which one wins
When a field exists at both levels and is set in both, the container one wins. This enables a very clean strategy: put the restrictive baseline on the pod and make specific, visible exceptions on the container that needs them.
spec:
securityContext:
runAsUser: 10001 # baseline for everyone
containers:
- name: app
# inherits runAsUser: 10001
- name: log-adapter
securityContext:
runAsUser: 10002 # this container overrides itOne confusing nuance: capabilities and allowPrivilegeEscalation are not inherited from the pod because they do not exist at pod level. They must be repeated in every container, including the initContainers and the sidecars. It is tedious and it is the most frequent reason why a "hardened" pod has one unhardened container. Kustomize or Helm help; an admission policy (08-03) guarantees it.
The initContainers count too
spec:
initContainers:
- name: prepare-data
image: registry.rutasnorte.example/utilities:1.4.2
securityContext: # it CANNOT be omitted
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
containers:
- name: app
# ...A privileged initContainer is a privileged pod: it runs with full access before the main container even starts. Go back over 06-04 and check that all your initContainers and sidecars are hardened just like the main containers.
- Running without root:
runAsNonRoot, runAsUser and runAsGroup
runAsNonRoot, runAsUser and runAsGroupBy default a container runs as whatever user the image declares; and a great many public images, nginx included, declare root (UID 0). Running as root inside the container is not the same as being root on the node, but it is far closer than anyone would like:
- Root inside the container has, by default, a set of capabilities that an ordinary user does not.
- If the container shares anything with the node (a
hostPath, a socket), root writes where an ordinary user could not. - When a kernel escape bug appears, most of them require being root inside the container to work. Not being root removes a good part of that class of problem.
The three fields
spec:
securityContext:
runAsNonRoot: true # a check: the kubelet refuses the pod if the UID is 0
runAsUser: 10001 # the process's effective UID
runAsGroup: 10001 # primary GID| Field | What it does exactly |
|---|---|
runAsNonRoot: true |
It does not change the user. It is a check: if the resulting UID is 0, the kubelet does not start the container |
runAsUser: N |
Forces the UID, ignoring the image's USER |
runAsGroup: N |
Forces the primary GID. If omitted, it usually ends up as 0 (the root group), which is not ideal |
runAsNonRoot without runAsUser works only if the image declares a numeric user. If the Dockerfile says USER appuser (a name, not a number), the kubelet cannot resolve the name —it has no access to the image's /etc/passwd before starting it— and the pod fails:
Error: container has runAsNonRoot and image has non-numeric user (appuser),
cannot verify user is non-rootHence a rule we will pick up again in 08-05: always declare the numeric user in the Dockerfile (USER 10001), and specify runAsUser in the manifest as well. Belt and braces.
If you try to run as root with the check enabled:
The nginx case in web-store
web-store uses the official nginx image, which starts as root so that it can listen on port 80 and then drops privileges in its worker processes. If we simply add runAsNonRoot: true, it fails when trying to write to /var/cache/nginx and when opening port 80.
There are two roads, and only one is good:
| Option | What it implies |
|---|---|
Add the NET_BIND_SERVICE capability for port 80 |
It works, but it keeps an unnecessary capability |
Use the nginxinc/nginx-unprivileged image and listen on 8080 |
No capabilities, no root, and the Service translates the port |
The second is clearly superior and it is the one we adopt:
# k8s/base/web-store/deployment.yaml (fragment)
spec:
template:
spec:
securityContext:
runAsNonRoot: true
runAsUser: 101 # the nginx user of the unprivileged image
runAsGroup: 101
fsGroup: 101
seccompProfile:
type: RuntimeDefault
containers:
- name: nginx
image: registry.rutasnorte.example/nginx-unprivileged:1.27.1
ports:
- containerPort: 8080 # high port: no capability requiredAnd the Service still publishes port 80 to the outside world:
apiVersion: v1
kind: Service
metadata:
name: web-store
namespace: rutas-norte-pro
spec:
selector:
app: web-store
ports:
- port: 80 # what clients see
targetPort: 8080 # what the unprivileged container listens onNobody outside the cluster notices the difference, and we have removed one capability and the root start-up.
- Volumes and file permissions:
fsGroup and fsGroupChangePolicy
fsGroup and fsGroupChangePolicyHere comes the practical problem that makes many people give up and go back to root. It is exactly the case of bookings-postgres.
The problem
A dynamically provisioned volume (module 5) is mounted with whatever permissions the file system gives it, normally owned by root:root with mode 0755. If the container runs as UID 10001, it cannot write to its own data volume.
The solution: fsGroup
When fsGroup is present, the kubelet, before starting the containers:
- Changes the owning group of the volume's files and directories to that GID.
- Adds the write bit for the group.
- Sets the
setgidbit on the directories, so that new content inherits the group. - Adds that GID to the process's supplementary groups.
The result: the process can write to the volume without being root and without touching the image.
fsGroup does not apply to every volume type. It works with those that have their own file system (most block volumes through CSI, emptyDir, configMap, secret) and not with NFS or other network file systems, where permissions are governed by the server. With NFS you have to coordinate UIDs/GIDs with whoever administers the storage.
fsGroupChangePolicy: the detail that matters in production
Changing the owner of every file is a recursive operation. On the bookings-postgres volume, with millions of files, that can take minutes on every pod start. And it happens on every restart, on every update, on every move between nodes.
spec:
securityContext:
fsGroup: 999
fsGroupChangePolicy: OnRootMismatch # <-- essential on large volumes| Value | Behaviour | When to use it |
|---|---|---|
Always (the default) |
Walks the whole volume on every start | Small volumes |
OnRootMismatch |
Checks only the permission of the volume's root directory; if it already matches, it does nothing | Large volumes: nearly always the right one |
OnRootMismatch works because, if the root directory already has the expected group and permissions, the change must have been made on a previous start. The first time it does the full walk; the following ones are instantaneous.
The complete bookings-postgres case
The postgres:16.4 image defines the postgres user with UID and GID 999. The final configuration:
# k8s/base/bookings-postgres/statefulset.yaml (fragment)
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: bookings-postgres
namespace: rutas-norte-pro
spec:
serviceName: bookings-postgres
template:
spec:
securityContext:
runAsNonRoot: true
runAsUser: 999 # the image's postgres user
runAsGroup: 999
fsGroup: 999 # the volume becomes writable
fsGroupChangePolicy: OnRootMismatch # without this, start-ups of several minutes
seccompProfile:
type: RuntimeDefault
containers:
- name: postgres
image: registry.rutasnorte.example/postgres:16.4
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: false # PostgreSQL writes to /var/run and /tmp
capabilities:
drop: ["ALL"]
env:
# PGDATA in a subdirectory: avoids problems with lost+found
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
volumeMounts:
- name: data
mountPath: /var/lib/postgresql/data
# Metrics exporter sidecar from module 7: hardened as well
- name: metrics-exporter
image: registry.rutasnorte.example/postgres-exporter:0.15.0
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 65534
capabilities:
drop: ["ALL"]
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: rutasnorte-ssd
resources:
requests:
storage: 50GiFour decisions worth justifying:
runAsUser: 999: the UID the image already expects. Using another one would force us to modify the image.fsGroupChangePolicy: OnRootMismatch: without it, with 50 GiB of personal data, every pod restart would take minutes and would look like a service outage.readOnlyRootFilesystem: falseon PostgreSQL: this is a conscious, documented exception. PostgreSQL writes socket and temporary files outside the data volume. It could be fixed with anemptyDiron/var/run/postgresqland/tmp, and that is the right move in the medium term; here we leave it explicit so you can see that a documented exception is better than a silent one.- The exporter sidecar does carry
readOnlyRootFilesystem: true: it writes nothing, so there is no reason to leave it open. Each container is hardened according to what it needs, not according to what its neighbour needs.
allowPrivilegeEscalation and the no_new_privs bit
allowPrivilegeEscalation and the no_new_privs bitThis line, which costs nothing, translates in the kernel to the process's no_new_privs bit (documented in prctl(2)). Its exact meaning:
With
no_new_privsset, no child process can gain more privileges than its parent, not even by executing a binary with thesetuidbit set or with file capabilities.
What a setuid binary is
In Linux, an executable can carry the setuid bit, which makes it run with the UID of its owner rather than that of whoever launched it. The classic example is /usr/bin/passwd: an ordinary user launches it, but it runs as root because it needs to write to /etc/shadow.
Inside a container, a root-owned setuid binary is a bridge between "I am user 10001" and "I am root in this container". Many base images ship several without anyone noticing:
With allowPrivilegeEscalation: false, running any of them does not raise privileges: the setuid bit is neutralised.
When it should be true
Practically never in an application. There are three legitimate situations:
| Situation | Why |
|---|---|
The container is privileged: true |
It would be contradictory; Kubernetes forces it to be true |
The container adds capabilities with CAP_SYS_ADMIN |
It usually needs the escalation |
| Tools that depend on setuid | Extremely rare in modern workloads |
A consistency note: allowPrivilegeEscalation: false is incompatible with privileged: true. If you set both, the pod is rejected. It is a deliberate check by Kubernetes.
A useful side effect
no_new_privs is also the condition for seccomp to work without special capabilities. Setting it to false prepares the ground for section 9.
Practical rule: allowPrivilegeEscalation: false in every container, with no exceptions that are not documented and approved.
privileged: true and why it is the same as handing over the node
privileged: true and why it is the same as handing over the nodeWhen a container is privileged:
| What gets disabled | Consequence |
|---|---|
| The capability trimming | The process has every Linux capability |
| The device restrictions | It sees the node's entire /dev, raw disks included |
| The default seccomp profile | It can make any system call |
| AppArmor / SELinux | No mandatory confinement |
| Mount restrictions | It can mount the node's file systems |
The practical consequence is easy to state:
A privileged container is, to all practical effects, root on the node. It can read and write the disks, load kernel modules, reach the host file system and, with it, the mounted secrets of every other pod on that node.
Applied to Rutas Norte: if web-store ran privileged and sat on the same node as bookings-postgres, whoever controlled web-store would have access to the personal data volume. And neither RBAC nor the NetworkPolicies of module 4 would prevent it, because it is not an API request nor network traffic: it is direct disk access.
The legitimate uses and how to reduce them
privileged: true has real uses, all of them at the infrastructure level, not in applications:
| Component | Why it needs it | Finer-grained alternative |
|---|---|---|
| CNI plugin (Calico, Cilium) | Configures the node's networking | NET_ADMIN + hostNetwork |
| CSI drivers | Mount volumes on the node | SYS_ADMIN + bidirectional mount |
| Security agents (Falco) | Observe system calls | SYS_PTRACE, BPF, PERFMON |
| Log collector (module 7) | Reads the node's /var/log |
Read-only hostPath, unprivileged |
That last one matters and we take advantage of it: the module 7 log collector does not need to be privileged. Mounting /var/log read-only is enough.
# k8s/base/logs/daemonset.yaml (fragment) — hardened version
containers:
- name: collector
image: registry.rutasnorte.example/fluent-bit:3.1.7
securityContext:
privileged: false # NOT needed
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsUser: 0 # it does need root to read the node's logs
capabilities:
drop: ["ALL"]
add: ["DAC_READ_SEARCH"] # only to bypass READ permissions
volumeMounts:
- name: varlog
mountPath: /var/log
readOnly: true # <-- read-only
volumes:
- name: varlog
hostPath:
path: /var/log
type: DirectoryIt is still a pod with more privileges than an ordinary application —we warned about that in 06-02— but we have come down from "full control of the node" to "reading one directory". That reduction is exactly the work of this lesson.
Rule: if a third-party manifest asks for privileged: true, demand a concrete justification before applying it. Very often it is a convenient default, not a necessity.
- Linux capabilities: drop everything and add only what is needed
Historically Linux had two classes of process: root (could do everything) and the rest (could do almost nothing). Capabilities split root's powers into about forty independent pieces that are granted separately.
The relevant capabilities:
| Capability | What it allows | Risk |
|---|---|---|
NET_BIND_SERVICE |
Listen on ports < 1024 | Low |
CHOWN |
Change file ownership | Low/medium |
DAC_OVERRIDE |
Bypass all file permissions | High |
DAC_READ_SEARCH |
Bypass read permissions | Medium |
SETUID / SETGID |
Change identity | Medium |
NET_RAW |
Raw sockets (ping, network scanning) | Medium |
NET_ADMIN |
Configure networking: interfaces, firewall, routes | Very high |
SYS_ADMIN |
A catch-all: mounting, namespaces... | Critical |
SYS_PTRACE |
Inspect other processes' memory | High |
SYS_MODULE |
Load kernel modules | Critical: that is the node |
SYS_TIME |
Change the system clock | Medium |
BPF, PERFMON |
eBPF programs and profiling | High |
A container with no special configuration does not start with all of them: the runtime grants a default set of about fourteen, among them CHOWN, DAC_OVERRIDE, NET_RAW, SETUID, SETGID and NET_BIND_SERVICE. No ordinary web application uses a single one.
The correct practice
securityContext:
capabilities:
drop: ["ALL"] # start from zero
# add: [...] # and add only what is strictly indispensabledrop: ["ALL"] followed by nothing is the target configuration for every Rutas Norte component. None of them needs a single capability.
An important detail: drop: ["ALL"] and add: ["NET_BIND_SERVICE"] together do work and are the correct way to express "only this one". The drop is processed before the add.
The NET_BIND_SERVICE example and its better alternative
Suppose you want to keep the official nginx image listening on 80, without being root:
# It works, but it is not the best option
securityContext:
runAsNonRoot: true
runAsUser: 101
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
add: ["NET_BIND_SERVICE"] # the only way to open port 80 without being rootLet us compare it with the alternative we already adopted:
With NET_BIND_SERVICE on 80 |
With port 8080 | |
|---|---|---|
| Capabilities | One | None |
Meets PSS restricted (08-03) |
Yes (it is the only permitted exception) | Yes |
| Changes to the image | None | nginx configuration or the unprivileged image |
| What the client sees | Port 80 | Port 80 (the Service translates it) |
The low port is a restriction from an era when servers were shared physical machines. In Kubernetes, where the Service translates ports at no cost, keeping it only adds an unnecessary capability. The general rule: if you can avoid a capability by changing a port number, change it.
Seeing the effective capabilities
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000000000000000
CapAmb: 0000000000000000All zeros: exactly what we want. To translate a non-zero value into names:
kubectl exec -n rutas-norte-pro deploy/bookings-api -- \
sh -c 'capsh --decode=$(grep CapEff /proc/1/status | cut -f2)'If you saw something like cap_chown,cap_dac_override,cap_net_raw,cap_setgid,cap_setuid, then the drop: ["ALL"] is missing from that container.
readOnlyRootFilesystem and how to make it workable
readOnlyRootFilesystem and how to make it workableThe container's root file system is mounted read-only. Explicitly mounted volumes remain writable.
Why it is so valuable
Without readOnlyRootFilesystem |
With readOnlyRootFilesystem: true |
|---|---|
A binary can be written into /usr/bin |
No |
| A loaded library can be modified | No |
| A configuration file can be overwritten | No |
| A change persists for the container's lifetime | Any change is impossible |
It turns the container into something immutable at runtime: what is deployed is what runs, from beginning to end. And it gives a very useful property for module 7: if something tries to write to /, an error is logged. That is a free detection signal, and in 08-06 we will turn it into a Falco rule.
The problem and its solution
Almost every piece of software writes something: temporary files, sockets, caches, PIDs. The solution is to mount an emptyDir exactly in those places.
web-store with nginx is the canonical example, because nginx needs four cache directories plus /tmp and its PID file:
# k8s/base/web-store/deployment.yaml (complete fragment)
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-store
namespace: rutas-norte-pro
labels:
app: web-store
app.kubernetes.io/part-of: rutas-norte
spec:
replicas: 3
selector:
matchLabels:
app: web-store
template:
metadata:
labels:
app: web-store
app.kubernetes.io/part-of: rutas-norte
spec:
automountServiceAccountToken: false # from 03-06: it does not talk to the API
securityContext:
runAsNonRoot: true
runAsUser: 101
runAsGroup: 101
fsGroup: 101
seccompProfile:
type: RuntimeDefault
containers:
- name: nginx
image: registry.rutasnorte.example/nginx-unprivileged:1.27.1
ports:
- containerPort: 8080
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true # <-- the goal
capabilities:
drop: ["ALL"]
volumeMounts:
# Everything nginx needs to write, in memory and ephemeral
- name: nginx-cache
mountPath: /var/cache/nginx
- name: nginx-run
mountPath: /var/run # the PID file goes here
- name: temp
mountPath: /tmp
resources:
requests: { cpu: 50m, memory: 64Mi }
limits: { cpu: 500m, memory: 128Mi }
volumes:
- name: nginx-cache
emptyDir:
medium: Memory # in RAM: faster and it never touches the disk
sizeLimit: 64Mi # a limit: an unbounded emptyDir can fill the node
- name: nginx-run
emptyDir:
medium: Memory
sizeLimit: 8Mi
- name: temp
emptyDir:
sizeLimit: 128MiPoints worth highlighting:
medium: Memorymakes theemptyDirlive in RAM (tmpfs). For small caches it is faster and avoids writing data to the node's disk, which matters if session information passes through them.sizeLimitis mandatory in practice. An unboundedemptyDircan grow until it exhausts the node's disk (or memory). Withmedium: Memoryit also counts against the pod's memory limit.- The
emptyDirvolumes are emptied when the container restarts, which is exactly what we want: nothing persists.
How to find out which directories are needed
Do not guess: measure. Deploy with readOnlyRootFilesystem: true in rutas-norte-dev and see what fails.
The log tells you the exact path. You add the emptyDir, repeat, and in two or three iterations you have the full list. It is a ten-minute procedure per component, done once.
Usual directories by application type:
| Application | Directories it usually needs |
|---|---|
| nginx | /var/cache/nginx, /var/run, /tmp |
Node.js (bookings-api) |
/tmp (and /home/node/.npm if it installs at runtime, which it should not) |
| Java | /tmp (the JVM writes its performance files there) |
| Python | /tmp; avoid .pyc with PYTHONDONTWRITEBYTECODE=1 |
| PostgreSQL | /var/run/postgresql, /tmp, in addition to the data volume |
| Redis | /data (a real volume, not an emptyDir) |
- Seccomp, AppArmor and SELinux
Seccomp: filtering the system calls
Secure Computing Mode limits which system calls the process can make. It is the most direct defence against kernel bugs: if a bug is in a call your container cannot make, it does not affect you.
type value |
What it does |
|---|---|
Unconfined |
No filter. Every call available |
RuntimeDefault |
The runtime's profile: blocks around 60 dangerous calls |
Localhost |
A custom profile in a file on the node (requires localhostProfile) |
Watch out for a historical detail: Kubernetes' default value was Unconfined for years. Since 1.27 the kubelet's SeccompDefault gate allows changing it to RuntimeDefault, but it must be enabled explicitly in the kubelet configuration. Until you do, if you do not set seccompProfile, your container has no seccomp filter at all. This is one of those things people take for granted and should not.
RuntimeDefault blocks calls such as mount, reboot, init_module, kexec_load, ptrace in some contexts, bpf and pivot_root. It is compatible with practically any ordinary application, so setting it is almost free and should always be done.
Custom profiles
When RuntimeDefault is not enough —for instance, for a highly exposed component such as bookings-api— you can write your own profile. The file goes into /var/lib/kubelet/seccomp/profiles/ on each node (usually via a DaemonSet or the Security Profiles Operator):
{
"defaultAction": "SCMP_ACT_ERRNO",
"defaultErrnoRet": 1,
"architectures": ["SCMP_ARCH_X86_64", "SCMP_ARCH_X86", "SCMP_ARCH_X32"],
"syscalls": [
{
"names": [
"accept4", "bind", "listen", "socket", "connect", "getsockname",
"read", "write", "readv", "writev", "close", "openat", "fstat",
"epoll_create1", "epoll_ctl", "epoll_pwait", "futex",
"mmap", "munmap", "mprotect", "brk", "rt_sigaction", "rt_sigprocmask",
"clock_gettime", "getpid", "gettid", "exit", "exit_group",
"nanosleep", "sched_yield", "madvise", "getrandom", "fcntl"
],
"action": "SCMP_ACT_ALLOW"
}
]
}spec:
securityContext:
seccompProfile:
type: Localhost
localhostProfile: profiles/bookings-api.json # path relative to the seccomp directorySerious warnings about custom profiles:
- They are fragile. An update to the standard library or the runtime can introduce a new call and the process dies with
SIGSYSor an incomprehensibleEPERM. - They have to be generated by observation, not written by hand. The Security Profiles Operator can record the profile of a running workload.
- Always start with
RuntimeDefault. Moving to a custom profile only pays off on very critical components and where there is capacity to maintain it.
For debugging you can use defaultAction: SCMP_ACT_LOG, which logs instead of blocking. Never leave that in production: it protects against nothing.
AppArmor
Path-based mandatory access control, common on Debian and Ubuntu. Since Kubernetes 1.30 it has a native field in the securityContext, with no need for annotations:
spec:
securityContext:
appArmorProfile:
type: RuntimeDefault
containers:
- name: app
securityContext:
appArmorProfile:
type: Localhost
localhostProfile: rutasnorte-api # profile loaded on the nodeThe container.apparmor.security.beta.kubernetes.io/<container> annotations still work for compatibility, but they are deprecated: use the field.
SELinux
Label-based mandatory access control, common on Red Hat and derivatives. It is configured with seLinuxOptions:
In practice, the most frequent use in Kubernetes is to let the runtime assign labels automatically. Changing them by hand is usually counterproductive unless the organisation has its own SELinux policy.
Comparison
| Seccomp | AppArmor | SELinux | |
|---|---|---|---|
| What it controls | System calls | File paths and capabilities | Labels on every object |
| Where it is common | Every distribution | Debian, Ubuntu, SUSE | RHEL, Fedora, CentOS |
| Difficulty | Low with RuntimeDefault |
Medium | High |
| Portability between clusters | High | Depends on the node | Depends on the node |
| Recommendation | RuntimeDefault always |
If your nodes ship it | If your organisation already uses it |
Recommendation for Rutas Norte: seccompProfile: RuntimeDefault on absolutely every pod, and AppArmor/SELinux according to what the nodes provide, without relying on them for the essential security (because the local cluster is minikube and the production one might not match).
- The pod settings to avoid
There are five settings that break the pod's isolation from the node. We have mentioned them along the way in 05-01 and 06-02; here we gather them together.
| Setting | What it breaks | Concrete risk in Rutas Norte |
|---|---|---|
hostNetwork: true |
The network namespace | The pod sees all the node's traffic and bypasses the NetworkPolicies of 04-06 |
hostPID: true |
The PID namespace | It sees the other pods' processes, with their command lines and their memory |
hostIPC: true |
The IPC namespace | It reaches the shared memory of other processes on the node |
hostPath |
The mount namespace | Reads or writes the node's disk, including other pods' mounted secrets |
hostPort |
The port allocation | Opens a port on the node's IP, bypassing the Service and the Ingress |
hostNetwork and its interaction with NetworkPolicies
This one deserves its own paragraph because it dismantles the work of an earlier module. In 04-06 we built a deny-all policy in rutas-norte-pro and then authorised each conversation one at a time. A pod with hostNetwork: true uses the node's network stack, not the pod's, so NetworkPolicies —which act on pod IPs— do not apply to it. All that work is voided for that pod.
It is the reason why a DaemonSet with hostNetwork must not share a namespace with the applications, and why its manifests deserve especially careful review.
hostPath: the warning from 05-01, expanded
# NEVER in production for an application
volumes:
- name: dangerous
hostPath:
path: / # the node's entire disk
type: DirectoryA hostPath to / gives access to:
/var/lib/kubelet/pods/*/volumes/kubernetes.io~projected/— the ServiceAccount tokens of every pod on the node./etc/kubernetes/— on a control plane node, the cluster's keys./var/lib/dockeror/var/lib/containerd— every image and layer./var/run/containerd/containerd.sock— the runtime socket, which allows starting privileged containers.
That last point is key and is often overlooked: mounting the runtime socket is equivalent to privileged: true, even if the pod does not declare it. Any manifest that mounts /var/run/docker.sock or containerd.sock must be treated as a privileged pod.
The legitimate uses of hostPath are infrastructure ones (reading /var/log for the collector, exposing devices to a CSI driver) and they must always be:
- The most specific path possible, never
/nor/var. - With
readOnly: truewhenever possible. - With an explicit
type:(Directory,File,Socket) so that it is not created by accident. - In a separate namespace with a relaxed security profile (08-03).
hostPort
It opens the port on the node's IP. The problems: only one pod per node fits with that port, it bypasses the Service and the Ingress (and therefore the TLS of 04-05 and the WAF of 08-04), and it exposes the service to anyone who can reach the node's IP. To publish services you use Ingress, as in module 4.
RuntimeClass and sandboxed runtimes
RuntimeClass and sandboxed runtimesEverything above hardens a container, but it does not change the fundamental fact: the kernel is shared. If you need strong isolation, something else is required.
A RuntimeClass selects which runtime the pod uses:
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: gvisor
handler: runsc # configured in containerd on the nodeapiVersion: v1
kind: Pod
metadata:
name: isolated-workload
spec:
runtimeClassName: gvisor # this pod uses gVisor instead of runc
containers:
- name: app
image: registry.rutasnorte.example/third-party-processor:1.2.0The options
| Runtime | How it isolates | Start-up cost | I/O cost | Compatibility |
|---|---|---|---|---|
runc (the default) |
Namespaces + cgroups | Minimal | Minimal | Total |
gVisor (runsc) |
A user-space kernel that intercepts the calls | +50-100 ms | Noticeable on I/O-heavy work | High, with exceptions |
| Kata Containers | A lightweight virtual machine per pod | +200-500 ms | Low with virtio |
Very high |
| Firecracker | Micro-VM (the basis of Kata and of several cloud services) | +125 ms | Low | High |
When it pays off
| Workload | Reinforced isolation? |
|---|---|
web-store, bookings-api, notifications-worker |
No. Our own reviewed code; ordinary hardening is enough |
bookings-postgres |
No. The I/O cost would be unacceptable; it is protected with RBAC, networking and encryption |
| Running code submitted by third parties | Yes, essential |
| Multi-tenancy with customers who do not know each other | Yes |
| Parsing a customer file with a complex library | Probably yes |
Rutas Norte has no workload today that justifies it. If tomorrow it let the bus companies upload a timetable file and processed it with a complex parsing library, that component would be the natural candidate: it runs code over data that comes from outside.
The important thing is to know the tool and its criterion: reinforced isolation is for code you do not trust, not for your own.
- The full hardening of Rutas Norte
We bring everything together in the final configuration of each component, with the justification for every exception.
Summary table
| Component | UID | RootFS RO | Caps | Seccomp | Exceptions |
|---|---|---|---|---|---|
web-store |
101 | Yes | drop: ALL |
RuntimeDefault |
3 emptyDir |
bookings-api |
10001 | Yes | drop: ALL |
RuntimeDefault |
emptyDir on /tmp |
bookings-postgres |
999 | No | drop: ALL |
RuntimeDefault |
Writable RootFS (documented) |
redis-cache |
999 | Yes | drop: ALL |
RuntimeDefault |
Volume on /data |
notifications-worker |
10002 | Yes | drop: ALL |
RuntimeDefault |
emptyDir on /tmp |
occupancy-reports |
10003 | Yes | drop: ALL |
RuntimeDefault |
emptyDir for the report |
| Payments ambassador | 10004 | Yes | drop: ALL |
RuntimeDefault |
None |
| Log collector | 0 | Yes | drop: ALL + DAC_READ_SEARCH |
RuntimeDefault |
Read-only hostPath |
bookings-api: the complete case
# k8s/base/bookings-api/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: bookings-api
namespace: rutas-norte-pro
labels:
app: bookings-api
app.kubernetes.io/part-of: rutas-norte
environment: pro
spec:
replicas: 4
selector:
matchLabels:
app: bookings-api
template:
metadata:
labels:
app: bookings-api
app.kubernetes.io/part-of: rutas-norte
environment: pro
spec:
serviceAccountName: bookings-api
automountServiceAccountToken: true # it does read a ConfigMap through the API (see 08-01)
securityContext:
runAsNonRoot: true
runAsUser: 10001
runAsGroup: 10001
fsGroup: 10001
seccompProfile:
type: RuntimeDefault
containers:
- name: api
image: registry.rutasnorte.example/bookings-api@sha256:9f2c1d4e8a7b6035c1e4d9a2b8f7c3e6d5a4b3c2e1f0a9b8c7d6e5f4a3b2c1d0
ports:
- name: http
containerPort: 8080
- name: metrics
containerPort: 9090
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
# Probes from module 7, unchanged
livenessProbe:
httpGet: { path: /health, port: http }
initialDelaySeconds: 10
periodSeconds: 10
readinessProbe:
httpGet: { path: /ready, port: http }
periodSeconds: 5
volumeMounts:
- name: temp
mountPath: /tmp
resources:
requests: { cpu: 200m, memory: 256Mi }
limits: { cpu: "1", memory: 512Mi }
# Ambassador container towards the external payment gateway (06-04)
- name: payments-ambassador
image: registry.rutasnorte.example/payments-ambassador:1.3.0
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 10004
capabilities:
drop: ["ALL"]
resources:
requests: { cpu: 20m, memory: 32Mi }
limits: { cpu: 100m, memory: 64Mi }
volumes:
- name: temp
emptyDir:
sizeLimit: 64MiNote the image referenced by digest instead of by tag. That is the subject of 08-05 and it fits here naturally: there is no point in hardening a container if you do not know for certain which image is running.
notifications-worker with its log adapter
# k8s/base/notifications-worker/deployment.yaml (fragment)
spec:
template:
spec:
automountServiceAccountToken: false
securityContext:
runAsNonRoot: true
runAsUser: 10002
runAsGroup: 10002
fsGroup: 10002
seccompProfile:
type: RuntimeDefault
containers:
- name: worker
image: registry.rutasnorte.example/notifications-worker:3.2.0
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: ["ALL"] }
volumeMounts:
- name: temp
mountPath: /tmp
- name: worker-logs
mountPath: /var/log/worker
# Adapter that converts the logs to JSON (06-04 and 07-05)
- name: log-adapter
image: registry.rutasnorte.example/log-adapter:1.1.0
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 10002 # same UID: it shares the log volume
capabilities: { drop: ["ALL"] }
volumeMounts:
- name: worker-logs
mountPath: /var/log/worker
readOnly: true # it only reads: there is no reason to write
volumes:
- name: temp
emptyDir: { sizeLimit: 64Mi }
- name: worker-logs
emptyDir: { sizeLimit: 256Mi }Two fine details: the adapter uses the same UID as the worker so that it can read the files the worker writes (alternative: a shared fsGroup), and it mounts the volume with readOnly: true because it only reads. Hardening a sidecar is just as important as hardening the main container: they share the network namespace and often volumes too.
The occupancy-reports CronJob
# k8s/base/occupancy-reports/cronjob.yaml (fragment)
apiVersion: batch/v1
kind: CronJob
metadata:
name: occupancy-reports
namespace: rutas-norte-pro
spec:
schedule: "0 3 * * *"
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
serviceAccountName: occupancy-reports
automountServiceAccountToken: false # it does not talk to the API (08-01)
securityContext:
runAsNonRoot: true
runAsUser: 10003
runAsGroup: 10003
fsGroup: 10003
seccompProfile:
type: RuntimeDefault
containers:
- name: reports
image: registry.rutasnorte.example/occupancy-reports:1.5.0
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: ["ALL"] }
volumeMounts:
- name: work
mountPath: /work
volumes:
- name: work
emptyDir: { sizeLimit: 512Mi }This component is particularly sensitive because it reads personal data in order to aggregate it. Remember what we said in 07-05: the reports contain only aggregates (occupancy per route and per time slot), never individual records, and none of it goes into the logs.
A reusable fragment
Since these blocks repeat, the sensible thing is to extract them. With Kustomize (module 10) a strategic patch is applied to every Deployment:
# k8s/base/common-hardening.yaml
# Reference block. Copy it or apply it as a patch.
podSecurityContext: &pod
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
containerSecurityContext: &container
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]And in 08-03 we will take the definitive step: making the cluster reject any pod that does not meet these rules, so that no future deployment can slip past them by oversight.
- How to check the result from inside the container
Unverified hardening is an assumption. These checks take a minute and should be done after every change.
Who am I
If you saw uid=0(root), then runAsUser is not being applied (check that the container's securityContext is not overriding it).
The root file system is read-only
kubectl exec -n rutas-norte-pro deploy/bookings-api -c api -- \
sh -c 'touch /write-test 2>&1 || echo "correct: blocked"'And that the emptyDir volumes really are writable:
kubectl exec -n rutas-norte-pro deploy/bookings-api -c api -- \
sh -c 'touch /tmp/ok && echo "correct: /tmp writable" && rm /tmp/ok'Effective capabilities
kubectl exec -n rutas-norte-pro deploy/bookings-api -c api -- \
grep -E 'CapEff|CapBnd|NoNewPrivs' /proc/1/statusCapEff: 0 confirms the drop: ["ALL"]. NoNewPrivs: 1 confirms allowPrivilegeEscalation: false. It is the most direct check of the two most important lines in the manifest.
Seccomp is active
Seccomp value |
Meaning |
|---|---|
0 |
Disabled: the profile is not being applied |
1 |
Strict mode (rare) |
2 |
Filter mode: RuntimeDefault or Localhost active |
If you see 0 when you expected RuntimeDefault, check that the field is at the right level and that the node supports it.
The namespace isolation works
Only the container's processes. If you saw node processes (kubelet, containerd, processes from other pods), then hostPID: true is enabled.
A complete verification script
#!/usr/bin/env bash
# k8s/security/verify-hardening.sh
# Checks the effective hardening of a running container.
set -uo pipefail
NS="${1:-rutas-norte-pro}"
WORKLOAD="${2:-deploy/bookings-api}"
CONTAINER="${3:-api}"
run_in() { kubectl exec -n "$NS" "$WORKLOAD" -c "$CONTAINER" -- "$@" 2>/dev/null; }
echo "=== $NS / $WORKLOAD / $CONTAINER ==="
echo -n "User: "; run_in id
echo -n "RootFS RO: "
if run_in sh -c 'touch /.test' >/dev/null 2>&1; then
echo "NO (writable) <-- REVIEW"; run_in rm -f /.test
else
echo "yes"
fi
echo -n "Capabilities: "
caps=$(run_in grep CapEff /proc/1/status | awk '{print $2}')
[[ "$caps" == "0000000000000000" ]] && echo "none (correct)" || echo "$caps <-- REVIEW"
echo -n "NoNewPrivs: "
nnp=$(run_in grep NoNewPrivs /proc/1/status | awk '{print $2}')
[[ "$nnp" == "1" ]] && echo "1 (correct)" || echo "$nnp <-- REVIEW"
echo -n "Seccomp: "
sec=$(run_in grep '^Seccomp:' /proc/1/status | awk '{print $2}')
[[ "$sec" == "2" ]] && echo "filter active (correct)" || echo "$sec <-- REVIEW"
echo -n "Processes seen: "; run_in sh -c 'ps aux | wc -l'=== rutas-norte-pro / deploy/bookings-api / api ===
User: uid=10001 gid=10001 groups=10001
RootFS RO: yes
Capabilities: none (correct)
NoNewPrivs: 1 (correct)
Seccomp: filter active (correct)
Processes seen: 3A practical note: this script needs pods/exec, a permission that in 08-01 we did not give to support or development. It is a platform task, and the sensible thing is to integrate it as an automatic check in rutas-norte-pre before promoting to production.
Common Mistakes and Tips
Putting capabilities or readOnlyRootFilesystem in the pod's securityContext. They do not exist at that level. The manifest is either rejected or the field is ignored, depending on the tool. They go on each container.
Forgetting the initContainers and the sidecars. A pod with a perfectly hardened main container and a privileged initContainer is a privileged pod. Go back over 06-04.
Using runAsNonRoot: true with an image that declares USER by name. The kubelet cannot verify it and the pod does not start. Always use a numeric UID in the Dockerfile (08-05) and runAsUser in the manifest.
Giving up on readOnlyRootFilesystem and removing it. The log tells you the exact path that fails. Two or three emptyDir volumes solve almost every case.
emptyDir without sizeLimit. It can fill the node's disk or memory and cause other pods to be evicted. With medium: Memory it also counts against the pod's memory limit.
Taking it for granted that seccomp is active. The default value was Unconfined for years. If you do not set seccompProfile: RuntimeDefault, there is probably no filter. Check it with grep Seccomp /proc/1/status.
Writing a custom seccomp profile by hand. It is fragile and breaks with every image update. Start with RuntimeDefault; if you need more, generate the profile by observing the workload.
Mounting the runtime socket. /var/run/docker.sock or containerd.sock are equivalent to privileged: true even if the manifest does not say so.
Accepting privileged: true in a third-party manifest without asking. Very often it is convenience, not necessity. Ask for the justification.
Forgetting fsGroupChangePolicy: OnRootMismatch on large volumes. The pod takes minutes to start on every restart and looks like an outage.
Believing that runAsNonRoot changes the user. It does not change it: it verifies it. Without runAsUser or a numeric USER in the image, it does not start.
Assuming a container isolates like a VM. It shares the kernel. That is the premise of the whole lesson and the reason nodes must be kept patched.
Golden tip: the target for every application is exactly this block, and any deviation must be commented in the YAML itself explaining why:
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsNonRoot: true
capabilities:
drop: ["ALL"]
seccompProfile:
type: RuntimeDefaultExercises
Exercise 1: harden redis-cache
redis-cache is a single-replica StatefulSet using the redis:7.4-alpine image, which runs as root and mounts a PVC on /data. Write the complete securityContext fragment (pod and container level) knowing that:
- The official Redis image defines the
redisuser with UID and GID 999. - Redis writes its data file to
/data(the PVC) and its PID file to/var/run/redis. - It needs no Linux capability whatsoever.
- The volume is 8 GiB.
State what would break if you omitted fsGroup and how you would detect it.
Exercise 2: diagnose a pod that will not start
After hardening notifications-worker, the pod ends up in CrashLoopBackOff:
Error: EACCES: permission denied, open '/app/queue/pending.dat'
at Object.openSync (node:fs:596:3)
at savePending (/app/lib/queue.js:44:18)The applied securityContext is:
spec:
securityContext:
runAsNonRoot: true
runAsUser: 10002
seccompProfile: { type: RuntimeDefault }
containers:
- name: worker
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: ["ALL"] }- What is the exact cause?
- Propose two solutions and argue which is better.
- Write the command that would confirm the diagnosis.
Exercise 3: audit the unhardened pods in the cluster
Write a command that lists every pod in rutas-norte-pro that breaches any of these conditions, stating which one:
readOnlyRootFilesystem: trueon all its containers.allowPrivilegeEscalation: falseon all its containers.capabilities.dropincludesALLon all its containers.- It does not use
hostNetwork,hostPIDorhostIPC. - No container is
privileged.
It must walk the initContainers as well.
Solutions
Solution 1
# k8s/base/redis-cache/statefulset.yaml (fragment)
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: redis-cache
namespace: rutas-norte-pro
spec:
serviceName: redis-cache
replicas: 1
template:
spec:
automountServiceAccountToken: false
securityContext:
runAsNonRoot: true
runAsUser: 999
runAsGroup: 999
fsGroup: 999 # makes the /data PVC writable
fsGroupChangePolicy: OnRootMismatch # 8 GiB: avoids slow walks
seccompProfile:
type: RuntimeDefault
containers:
- name: redis
image: registry.rutasnorte.example/redis:7.4-alpine
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
volumeMounts:
- name: data
mountPath: /data # PVC: writable through fsGroup
- name: run
mountPath: /var/run/redis # emptyDir: the PID file
- name: temp
mountPath: /tmp
resources:
requests: { cpu: 100m, memory: 256Mi }
limits: { cpu: 500m, memory: 512Mi }
volumes:
- name: run
emptyDir: { medium: Memory, sizeLimit: 8Mi }
- name: temp
emptyDir: { sizeLimit: 32Mi }
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: rutasnorte-ssd
resources:
requests:
storage: 8GiWithout fsGroup: the PVC is mounted owned by root:root with mode 0755. Redis, running as UID 999, cannot write to /data. The pod starts and dies as soon as it tries to persist:
Detection:
kubectl logs -n rutas-norte-pro redis-cache-0
kubectl exec -n rutas-norte-pro redis-cache-0 -- ls -ld /dataWith fsGroup: 999 applied:
The group becomes 999, the group write bit appears and the s indicates the setgid on the directory.
An additional note: since it is a cache, losing the volume is not critical. But it is critical that it does not start, because bookings-api would respond more slowly and seat availability could degrade.
Solution 2
1. The cause. readOnlyRootFilesystem: true prevents writing anywhere in the root file system that is not a mounted volume. The worker tries to write to /app/queue/pending.dat, which is inside the image and therefore on the read-only rootfs. The EACCES message on a write openSync is the typical signature of this problem. It is not a UID problem: even running as root, / would still be read-only.
2. Two solutions:
Option A — mount an emptyDir on /app/queue:
containers:
- name: worker
volumeMounts:
- name: queue
mountPath: /app/queue
volumes:
- name: queue
emptyDir: { sizeLimit: 256Mi }Option B — change the worker's configuration so that it uses /tmp, already mounted:
Which is better: it depends on whether that data must survive. And here there is a business consideration that goes beyond the YAML: /app/queue/pending.dat holds notifications waiting to be sent. With either option, that data is lost when the container restarts, because emptyDir is ephemeral. If a customer buys a ticket and the worker restarts before sending the email, that customer never gets their confirmation.
The correct solution, therefore, is C: notifications-worker should not have a local queue at all. The queue must live in a shared, durable store —Redis with persistence, or the database itself— so that any replica can pick the work up. That also fixes a problem that existed before any hardening: with several replicas, each one had its own local queue, invisible to the others.
As an immediate unblocking measure, A with an emptyDir is preferable to B, because it leaves the problem visible in the manifest (volumes: queue) instead of hiding it in an environment variable. And a ticket must be opened for C.
This exercise illustrates something important: hardening a container often uncovers pre-existing design flaws. The readOnlyRootFilesystem did not create the problem; it made it visible.
3. Confirming the diagnosis:
# Start temporarily without readOnlyRootFilesystem in rutas-norte-dev and check
kubectl exec -n rutas-norte-dev deploy/notifications-worker -c worker -- \
sh -c 'ls -ld /app/queue; touch /app/queue/test && echo WRITABLE || echo BLOCKED'The directory permissions are correct (owner 10002, the same UID as the process), and it still fails: that rules out a permissions problem and confirms that the cause is the read-only mount.
Solution 3
#!/usr/bin/env bash
# k8s/security/audit-hardening.sh
# Lists the pods that breach the hardening baseline.
set -euo pipefail
NS="${1:-rutas-norte-pro}"
kubectl get pods -n "$NS" -o json | jq -r '
.items[] as $pod
| ($pod.metadata.name) as $name
| [
# Pod-level settings
(if $pod.spec.hostNetwork == true then "hostNetwork" else empty end),
(if $pod.spec.hostPID == true then "hostPID" else empty end),
(if $pod.spec.hostIPC == true then "hostIPC" else empty end),
(if ($pod.spec.volumes // [] | map(select(.hostPath)) | length) > 0
then "hostPath" else empty end),
# Per-container settings, including the initContainers
( (($pod.spec.containers // []) + ($pod.spec.initContainers // []))[]
| . as $c
| (
(if ($c.securityContext.privileged // false) == true
then "privileged:\($c.name)" else empty end),
(if ($c.securityContext.readOnlyRootFilesystem // false) != true
then "rootfs-writable:\($c.name)" else empty end),
(if ($c.securityContext.allowPrivilegeEscalation // true) != false
then "escalation-allowed:\($c.name)" else empty end),
(if (($c.securityContext.capabilities.drop // []) | index("ALL")) == null
then "no-drop-ALL:\($c.name)" else empty end)
)
)
] as $problems
| if ($problems | length) > 0
then "\($name)\n " + ($problems | join("\n "))
else empty
end'How to read this particular output:
bookings-postgres: this is the documented exception from section 12. It is justified, although the task of resolving it with anemptyDiris still open.fluent-bit-collector: the logging DaemonSet with a read-onlyhostPath. Legitimate and already reduced to the minimum (noprivileged), but in 08-03 we will move it to a separate namespace so that it does not relax the profile of the whole ofrutas-norte-pro.
The fact that no other pod appears is the evidence that the hardening is applied. This script should run in the pipeline against rutas-norte-pre before every promotion to production, with a list of known exceptions so that it only alerts on new ones.
A note on the // true and // false in the jq: the default values are written in the unsafe direction on purpose. An absent allowPrivilegeEscalation means allowed, so its default is true; an absent readOnlyRootFilesystem means writable, so its default is false. An audit must assume the worst when the field is missing.
Conclusion
We have added the second layer of defence. Going over the essentials:
- A container is not a virtual machine. It shares a kernel with the node, and the isolation comes from namespaces, cgroups and the limits on what it can ask the kernel for. That is why patching the nodes is a top-tier security task.
- The
securityContextlives at two levels: identity and volumes on the pod; capabilities, escalation and file system on the container. The container's wins, andcapabilities,privileged,readOnlyRootFilesystemandallowPrivilegeEscalationmust be repeated on every container, initContainers and sidecars included. - Do not run as root (
runAsNonRoot+ a numericrunAsUser), and solve the volume permissions withfsGroupandfsGroupChangePolicy: OnRootMismatchwhen the volume is large, as inbookings-postgres. allowPrivilegeEscalation: falsesetsno_new_privsand neutralises setuid binaries. It costs one line.privileged: trueis the same as handing over the node, and mounting the runtime socket is too, even if it is not declared.capabilities: drop: ["ALL"]and add only what is indispensable; and very often evenNET_BIND_SERVICEcan be avoided by moving to a high port.readOnlyRootFilesystem: truemakes the container immutable at runtime, and it becomes workable with a handful ofemptyDirvolumes withsizeLimit.seccompProfile: RuntimeDefaultalways; do not take it for granted that it is already active.- Avoid
hostNetwork,hostPID,hostIPC,hostPathandhostPort; remember thathostNetworkvoids the NetworkPolicies of module 4. - For code you do not trust,
RuntimeClasswith gVisor or Kata; for your own, it does not pay off. - And check it all from the inside:
id, the failed write,CapEff,NoNewPrivsandSeccompin/proc/1/status.
Every Rutas Norte component is now hardened. But look at the nature of what we have done: we have written a correct YAML. Nothing stops somebody tomorrow from deploying a new pod without a securityContext, or copying a manifest from the internet with privileged: true, or a colleague adding a sidecar and forgetting the drop: ["ALL"]. All the hardening in this lesson depends on each person remembering, every time. And that is not a guarantee: it is a hope.
The next lesson, 08-03, Pod Security Policies and Pod Security Standards, takes the definitive step: moving from "each team sets its securityContext properly" to "the cluster does not accept a pod that lacks it". We will see the Pod Security Standards, the Pod Security Admission built into the apiserver, and policy engines such as Kyverno that let you enforce any rule you can think of, with the exact rejection message the API returns when a pod does not comply.
Kubernetes Course
Module 1: Introduction to Kubernetes
- What Is Kubernetes?
- Kubernetes Architecture
- Key Concepts and Terminology
- Setting Up a Kubernetes Cluster
- The Kubernetes CLI: kubectl
- Objects, YAML Manifests and the Declarative Model
- The Course Project: the Rutas Norte Platform
Module 2: Core Kubernetes Components
- Pods
- ReplicaSets
- Deployments
- Updates, Rollbacks and Deployment Strategies
- Services
- Namespaces
- Labels, Selectors and Annotations
Module 3: Configuration and Secret Management
- ConfigMaps
- Secrets
- Environment Variables
- Resource Quotas and Limits
- LimitRanges and Quality of Service (QoS) Classes
- ServiceAccounts and API Access from Pods
Module 4: Networking in Kubernetes
- Cluster Networking
- Service Types
- Internal DNS and Service Discovery
- Ingress Controllers
- TLS and Certificate Management with cert-manager
- Network Policies
Module 5: Storage in Kubernetes
- Volumes
- Persistent Volumes
- Persistent Volume Claims
- Storage Classes
- Dynamic Provisioning, Expansion and Snapshots
- Backup and Restore of Persistent Data
Module 6: Advanced Kubernetes Concepts
- StatefulSets
- DaemonSets
- Jobs and CronJobs
- Init Containers, Sidecars and Multi-Container Patterns
- Scheduling: Affinity, Taints and Tolerations
- Custom Resource Definitions (CRDs)
- Operators and the Controller Pattern
Module 7: Monitoring and Logging
- Health Checks and Probes
- Metrics Server and kubectl top
- Monitoring with Prometheus
- Visualization and Alerting with Grafana and Alertmanager
- Centralized Logging with Elasticsearch, Fluentd and Kibana (EFK)
- Application Debugging and Cluster Events
Module 8: Kubernetes Security
- Role-Based Access Control (RBAC)
- Security Contexts and Container Hardening
- Pod Security Policies and Pod Security Standards
- Network Security
- Image Security
- Auditing, Scanning and Vulnerability Management
Module 9: Scaling and Performance
- Horizontal Pod Autoscaling
- Vertical Pod Autoscaling
- Cluster Autoscaling
- Event-Driven and Custom-Metric Scaling with KEDA
- High Availability: PodDisruptionBudgets and Topology
- Performance Tuning
Module 10: Kubernetes Ecosystem and Tooling
- Minikube and Local Environments with kind
- Kubeadm
- Helm
- Kustomize
- GitOps with Argo CD and Flux
- Managed Kubernetes: EKS, AKS and GKE
Module 11: Case Studies and Real-World Applications
- Deploying a Web Application
- Running Stateful Applications
- CI/CD with Kubernetes
- Deployment Strategies: Blue-Green and Canary
- Multi-Cluster Management
- Production Operations: Incidents, Runbooks and Costs
