You have spent six modules using containers. This lesson explains why they work. The thesis is uncomfortable and liberating at the same time: a container does not exist. There is no "container" entity in the Linux kernel. There are ordinary processes with three mechanisms applied on top, and every one of them can be handled by hand.

1. The thesis: three mechanisms and no object
2. The seven namespaces
3. The same process, two different PIDs
4. /proc/<pid>/ns/, lsns and comparing containers
5. Getting in by hand with nsenter
6. Creating a namespace from scratch with unshare
7. The user namespace and UID remapping
8. Cgroups v2: the hierarchy
9. aurora-db's limits, read from the kernel
10. The OOM killer through memory.events
11. OverlayFS: the real mount
12. Copy-on-write in the upperdir
13. Capabilities and seccomp at kernel level
14. What runc actually does
15. Docker Desktop: all of this happens in a VM

When you run docker run, no magical structure is created. An ordinary process is launched and three things are applied to it:

Everything else —images, registries, Compose, healthchecks— is machinery built around those three kernel primitives.

A namespace is a partial view of a global system resource. The processes in a namespace see their version of the resource and not anybody else's.

An eighth one, time (the monotonic clock), has existed in the kernel since 5.6 but Docker does not use it yet.

This is the demonstration that convinces people fastest.

It is the same process. Inside it is PID 1 —hence everything you learned in lesson 03-02 about signals and PID 1—; on the host it is 48122, a child of the containerd shim and running as user 1000. There is no virtual machine: there is a node in your process list that happens to see a different tree from yours.

NSpid says it all: 48122 in the host's namespace, 1 in its own. One process with two simultaneous identities.

Each namespace is a special file whose inode identifies it:

Comparing two containers is comparing those numbers:

The most revealing result is the last one: aurora-api and aurora-db share the user namespace, and it happens to be the host's. Without --userns-remap or rootless mode, UID 0 inside is literally UID 0 on the host. There it is, measured: the claim from lesson 05-03 about why root in a container is dangerous.

docker exec is not magic: it enters the process's namespaces and launches a command. nsenter does the same thing, and without going through Docker.

The flags are exactly the namespaces: -m (mnt), -u (uts), -i (ipc), -n (net), -p (pid). You can enter only some of them, and that is what makes the technique useful: -n on its own gives you the container's network with the host's tools —precisely what you were doing in lesson 05-01— and it works even with distroless images that do not have an sh inside.

To see that there is no magic, let's build one by hand.

No Docker, no images and no daemon: a shell that believes it is PID 1, with its own host name and an empty network stack. That is a container's isolation. What Docker adds on top is everything else: the image's filesystem, the network configuration with its veth and its bridge, the cgroups, the capabilities, seccomp and an API to manage it all.

The user namespace lets the same UID mean different things inside and outside. It is the basis of rootless mode (lesson 05-03).

Inside you are root (id -u = 0) without having used sudo; the map says that UID 0 inside corresponds to 1000 outside, with length 1. Outside you are still 1000.

Root inside, but the kernel checks the file's permissions against the real host UID. That is the exact mechanism that makes the escalation from lesson 05-03 stop working in rootless mode: it is not a check Docker adds, it is UID arithmetic in the kernel.

Control groups limit and account for resources. Version 2, the unified one, has been the standard since 2022 and is exposed as a directory tree under /sys/fs/cgroup.

The essential difference from v1: in v2 a process belongs to a single cgroup and every controller acts on that one, instead of a separate hierarchy per resource. That simplifies the reasoning enormously.

Now the check that closes the circle with lesson 03-07 and with Compose: what you declared in YAML, read straight from the kernel.

Line by line: memory.max is 2,147,483,648 bytes = 2 GiB, the limits: { memory: 2G } from compose.prod.yaml; memory.current is what docker stats shows; pids.max is pids_limit: 200; and cpu.max with 200000 100000 means 200 ms of CPU for every 100 ms period, that is, cpus: "2.0".

An exact match. docker stats calculates nothing: it reads these files. And cpu.max finally explains the semantics of --cpus: it is not a number of assigned cores, it is a time quota per period; with 2.0, the container can consume 200 ms of CPU every 100 ms, spread across however many cores there are.

The counters tell the whole story: max 412 is the number of times the process hit its limit and the kernel had to reclaim memory; oom 1 says there came a point where it could reclaim no more; oom_kill 1 is the execution. The outcome, exit code 137, which you have known since lesson 03-02, is simply 128 + 9: terminated by SIGKILL.

A high max with oom_kill 0 is an extremely valuable signal that almost nobody watches: the container has not died, but it is fighting its limit and paying for it in latency. It is exactly what the MemoryNearLimit alert from lesson 05-06 catches before it is too late.

The container's root / is an overlay mount, with the four directories from lesson 05-02. And the lowerdirs correspond one to one with the image's layers:

Three layers in the image manifest, three lowerdirs in the mount. The correspondence is literal: every image layer is a directory on the host's disk, and OverlayFS stacks them in order.

The three operations, with their three distinct effects in the upperdir:

• A new file: it appears as it is. That is the A in docker diff.
• A modified file: the kernel performs a copy-up —copying the whole original from the lowerdir— and writes on top of it. That is the C, and it is the expensive operation you measured in lesson 05-02.
• A deleted file: a character device 0/0 appears, the famous whiteout. It deletes nothing: it hides the file from the lower layer, which still takes up space. That is the D, and it is the reason why deleting in a later layer does not slim down an image (lesson 05-04).

There it is, in three lines of ls: the physical explanation of lessons 01-05, 05-02 and 05-04.

The capabilities from lesson 05-03 are a bitmask in the process descriptor.

CapEff at zero: aurora-api runs with no effective capabilities at all, exactly as you asked for with cap_drop: [ALL]. Compare it with a default container:

Docker grants fourteen capabilities by default. The difference between that list and a CapEff of zero is the whole surface you have removed, and now you see it for what it is: bits in /proc.

Seccomp: 2 means BPF filter mode is active (0 would be disabled); NoNewPrivs: 1 is the no-new-privileges:true from compose.prod.yaml. Everything you configured in YAML ends up as two numbers in a file under /proc.

The chain from lesson 01-03, now seen from the inside: dockerd → containerd → shim → runc → process.

runc is a small binary that receives a directory with two things: the already-mounted filesystem and a config.json carrying the container's OCI specification.

You recognize every line: they are your docker run and Compose options translated into the standard. runc's steps are: create the namespaces (clone with the CLONE_NEW* flags), write the cgroups, pivot the root to rootfs (pivot_root), apply capabilities and seccomp, and finally execve the command. That is where its job ends: runc exits, and the process is left orphaned under the shim.

That is the role of the shim (containerd-shim-runc-v2): keeping stdin/stdout/stderr open, reporting the exit code and —crucially— allowing dockerd to restart without killing your containers, because the parent is not the daemon but the shim.

The fact that all of this is an open standard (the OCI Runtime Specification) is what lets you swap runc for crun, youki or gVisor without changing anything else. That ecosystem is lesson 07-05.

If you are on macOS or Windows, none of the above happens in your operating system: namespaces and cgroups are exclusively Linux kernel mechanisms. Docker Desktop runs a lightweight Linux virtual machine and the daemon lives inside it.

To reproduce this lesson from macOS or Windows, get into the VM:

And this explains earlier lessons along the way: the slowness of bind mounts (05-02 and 04-07) is not Docker's doing, it is the boundary between two filesystems; and that is why paths and processes "do not show up" where you expect them.

Believing a container is a lightweight machine. It is a process with three mechanisms on top and one single shared kernel. That is where the whole of lesson 05-03 comes from.

Looking for cgroups v1 on a modern system. Since 2022 it has been unified v2: one cgroup per process and different files (memory.max, not memory.limit_in_bytes). And do not write directly into /sys/fs/cgroup: Docker overwrites it when it reconciles; use docker update or Compose.

Running nsenter without -p and expecting to see the isolated process tree. Each namespace is entered separately; without -p you will see the host's processes.

Assuming that an oom_kill 0 means everything is fine. A high max indicates constant memory pressure and latency, even though nobody has died.

Trying lsns or /sys/fs/cgroup from macOS. They do not exist outside Linux. Get into Docker Desktop's VM.

Tip: when something does not add up, go down a level. A limit that seems not to apply is checked in memory.max; a networking problem, in /proc/<pid>/ns/net and nsenter -n; a file that appears or disappears, in the upperdir. The kernel does not lie and it can always be queried.

Exercise 1. Prove that a container is a host process: find aurora-api's real PID, check that inside it is PID 1, verify with NSpid that it is the same process, and enter its network namespace with nsenter without using docker exec.

Exercise 2. Check that Compose's limits are exactly the cgroup files: read memory.max, cpu.max and pids.max for aurora-db, translate them back into the options that produced them, change them with docker update and verify that the file changes on the fly.

Exercise 3. Trigger the three copy-on-write operations (create, modify and delete) in aurora-api and locate their three distinct effects in the upperdir. Explain which of the three explains why deleting files does not slim down an image.

Solution 1.

NSpid: 48122 1 is the definitive proof: a single process with two identities, the host namespace's and its own. There is no copy, no emulation and no virtual machine; there is a mapping table in the kernel.

And nsenter demonstrates the second point: you have obtained exactly what docker exec gives you without talking to the daemon at any moment. That has two practical consequences. The first is diagnostic: if the daemon is hung or the image is distroless and has no sh, nsenter still works because it uses the host's binaries against the container's namespaces. The second is about security: anybody with root on the host gets into any container without leaving a trace in Docker's logs, which reinforces why root access to the host is the boundary that really matters.

Solution 2.

The change is instant and requires no container restart: cgroups v2 lets you rewrite the limit on the fly, and the process does not even notice until it tries to exceed it. docker stats reflects the new value because, literally, it reads that file.

The underlying reading is that docker run --memory and deploy.resources are not Docker abstractions: they are a convenient way of writing a number into /sys/fs/cgroup. And the interpretation of cpu.max clears up the most common confusion in the course: --cpus 1.5 does not reserve one and a half cores, it grants 150 ms of CPU time every 100 ms period, which can be spread across the host's eight cores.

Solution 3.

Three operations, three physically distinct representations in the upperdir:

The third one explains the phenomenon from lesson 05-04. Deletion removes nothing from the lowerdir —it is read-only, it cannot—: it creates a special marker that makes OverlayFS hide the file when it resolves the unified view. Inside the container, ls answers No such file or directory; on disk, the original file is still intact in its layer, downloaded on every docker pull and stored on every node.

Hence the rule you applied with the &&, which now has its full explanation: if you create and delete inside the same RUN, both operations happen before the layer is consolidated, and the file never comes to exist in anybody's lowerdir. And that is why a credential written and deleted in separate instructions is still extractable with docker save: it is not a Docker bug, it is how a union filesystem works.

And look at the second row, which has consequences of its own: modifying one byte stores the entire file in the upperdir. With package.json that is 612 bytes; with a PostgreSQL data file it would be hundreds of megabytes. That, measured in lesson 05-02 and explained here, is the physical reason databases use volumes.

A container does not exist. What exists is an ordinary Linux process —you have seen it with its real PID in the host's ps and its NSpid carrying two identities— to which the kernel applies three mechanisms: namespaces for what it sees, cgroups for what it consumes and OverlayFS for the filesystem it has. You know the seven namespaces and which Docker option leans on each one, you know how to read them in /proc/<pid>/ns/, compare them with lsns, enter them by hand with nsenter without going through the daemon, and create one from scratch with unshare, confirming that the isolation is nothing magical. And you have discovered, by comparing inodes, that without rootless or --userns-remap all your containers share the host's user namespace: the exact explanation of why root inside is root outside.

You have read your own limits in the kernel: memory.max with the 2 GiB from compose.prod.yaml, pids.max with the 200, and cpu.max with 200000 100000, which reveals that --cpus 2.0 is a time quota per period and not a reservation of cores. You have changed them on the fly with docker update and watched the file move. You have seen the OOM killer from the inside, in memory.events, with its oom_kill 1 and the exit code 137 you already knew. And you have handled copy-on-write with your own hands: a new file, a complete copy for modifying a single byte, and a character device 0/0 for a deletion, which is the physical explanation of why deleting files neither slims down an image nor removes a secret. You close the security circle by seeing CapEff at zero decoded with capsh, Seccomp: 2 and NoNewPrivs: 1; and you know what runc does with its OCI-specification config.json and why the shim allows dockerd to restart without killing anything. Plus the essential note for anybody working on macOS or Windows: all of this happens inside a Linux VM, and that is where those paths and performance figures that did not add up were coming from.

And with that, module 5 closes. You came in knowing how to use Docker very well and you leave knowing exactly what happens when you run it. You have opened up networking down to the bridges, the veth pairs and the iptables rules, with the warning that publishing a port bypasses the host firewall; storage down to overlay2 and a backup strategy with verified restores; security down to the threat model, rootless mode, capabilities, seccomp, CVE scanning and Cosign signing, with Aurora Libros hardened in an annotated compose.prod.yaml; optimization, which took the API from 142 MB to 102 MB with multi-stage builds and constant measurement; BuildKit and Buildx, with cache mounts, traceless secrets, remote cache and multi-architecture; observability, with structured logs, Loki, Prometheus, ratio-based alerts and a /health that tells the truth; and, finally, the kernel that holds it all up.

In module 6, Aurora Libros leaves your machine. You will prepare the definitive production image, set up a CI/CD pipeline that builds, scans, signs and publishes using the remote cache you already know how to use, and take the platform to a cluster: first with Docker Swarm and its overlay networks, then with Kubernetes —its objects, the real deployment of the four services, scaling and load balancing— and you will finish with the deployment and rollback strategies that let you publish a new version without anybody losing the ability to buy a book.