The previous lesson ended with a weighty objection: a virtual machine isolates very well, but to achieve that it carries an entire kernel, an init, a complete file system, between 200 and 500 MB of RAM just to exist and tens of seconds of boot time. If what you want is to run your application with its dependencies, without it seeing the rest of the system and without it eating the machine, you are paying for an operating system you do not need.

Containers answer that with a radical change of perspective, and this is the sentence to take away from the whole lesson: a container is not a small virtual machine; it is an ordinary process of the host kernel whose view and consumption have been limited. There is no hypervisor, no guest kernel, no hardware emulation. There is a fork and an exec like the ones from 02-01, with three added ingredients: namespaces that lie to it about what exists, control groups that bound how much it can use, and a set of security restrictions — capabilities, seccomp, AppArmor — that you already know from Module 5.

Here you are going to see the three ingredients from the inside, one by one and with demonstrations you can run. Then you will really understand overlayfs, which is what turns an image into something shareable among a hundred containers, and we will close by packaging meteo-api into an image with a Dockerfile commented line by line. Orchestration and the cloud are the next lesson; here we stay on the machine.

1. The central idea: limited processes, not small machines
2. Namespaces, one by one
3. Real namespace inspection in /proc
4. cgroups v2: the unified hierarchy
5. The controllers in practice: cpu, memory, io and pids
6. How systemd uses cgroups
7. The third ingredient: security inside the container
8. Layered file systems: overlayfs for real
9. What an image is and what a container is
10. Runtimes, OCI standards and the real stack
11. Hands-on: packaging meteo-api
12. Containers versus virtual machines
13. Concrete risks and good practices

Let us start with a check that dismantles the wrong intuition. If on a Linux host you start a container and then look at the processes from outside:

What it proves. There it is: sleep is a host process, with PID 8123 in the host's process table, scheduled by the host's CFS-EEVDF, with its pages managed by the host's memory manager. There is no intermediate kernel. If you run kill 8123 from outside, the container dies. If the host panics, the container dies. Compare that with a VM, where from outside you would only see a qemu-system-x86 and never the processes inside.

The difference from an ordinary process lies only in what that process perceives:

From that follow the three consequences that govern everything else: instant startup, because there is no kernel to boot, only a clone() and an execve() (milliseconds, not seconds); very high density, because with no guest kernel and no init the baseline cost is a few megabytes; and weaker isolation, because the kernel is shared and the boundary becomes the system call table — 300 to 400 entry points — versus the handful of VM exit handlers from 06-01.

A namespace wraps a global system resource so that the processes inside see their own instance of that resource. Linux has eight. We already mentioned them in 03-02 as the CLONE_NEW* flags of clone(); now we open them up.

There are three system calls involved: clone() with the CLONE_NEW* flags creates a process already inside new namespaces, unshare() moves the current process into new namespaces, and setns() puts a process into an existing namespace (which is what docker exec does). The unshare tool wraps the first two, and all the examples that follow can be run.

It was the first one (Linux 2.4.19, year 2002) and it is the one that gives the container its own file system. It picks up 04-03 directly: the mount table is not global, each process sees its own, and /proc/<pid>/mountinfo tells you which.

What has happened. The mount really exists, but only inside that namespace. When you leave the shell, the namespace disappears and the mount with it, with no need to unmount anything. This is the mechanism that lets a container mount whatever it wants without dirtying the host.

Having your own mount table is not enough, though: you also have to change the root. Here it is worth distinguishing three things that get confused:

chroot() changes the / of that process, but the previous file system is still mounted and reachable: it is not security. pivot_root() changes the root of the mount namespace and allows the old one to be unmounted: that one is, combined with mnt. And a bind mount exposes a subtree at another point with different options: it is a tool, not a barrier.

About chroot you have to be blunt, because this is a classic mistake: chroot was never a security mechanism, and its own manual page says so. A process with CAP_SYS_CHROOT escapes with a recipe known since the 1990s — open a directory, chroot into a subdirectory and then climb up with chdir("..") from the open descriptor, because the kernel only checks the root on absolute paths — and also by creating a device node with mknod to read the raw disk, or with ptrace on a process outside.

That is why containers use pivot_root inside their own mount namespace, and then unmount the old root: at that point no reference to the host's file system remains, not because it is forbidden, but because it is no longer mounted in that namespace. That is the difference between hiding something and removing it.

It gives the container its own PID table. The first process inside is PID 1.

Why all three options are needed, which is the question everybody asks: --pid creates the new namespace; --fork is mandatory because whoever calls unshare() does not enter it, only its children do, so without --fork the shell would stay in the old namespace; and --mount-proc remounts /proc — which also implies a new mount namespace — because without that ps would keep reading the host's /proc and would show all the processes: ps does not ask the kernel, it reads files.

That --mount-proc detail is very instructive: it shows that PID isolation is provided by the kernel, but the view you get depends on which /proc you are reading. A badly built container that mounts the host's /proc reveals every process on the machine.

And here a topic from 02-01 reappears: zombies. The kernel imposes two special responsibilities on PID 1. First, adopting orphans and calling wait() on them to free their entry in the process table: if the container's PID 1 is your application and it does not call wait(), zombies accumulate until the PID limit is exhausted. And second, handling signals, because default actions do not apply to PID 1: if it does not install a SIGTERM handler, the signal is ignored. The practical consequence is that docker stop sends SIGTERM, the process ignores it, and 10 seconds later a SIGKILL arrives: the container always takes 10 seconds to stop and never closes its files cleanly.

The standard solution is to use a minimal init as PID 1 (tini, dumb-init, or --init in Docker), which forwards signals and reaps zombies, leaving the application as PID 2.

It gives the container its own network stack: interfaces, addresses, routing tables, nftables rules and ports. It is the reason a hundred containers can all listen on port 443 without colliding.

What it does, step by step. A veth pair is a virtual cable with two ends: what goes in one end comes out the other. Both are created on the host and then one is moved into the container's namespace. From then on they are two machines connected by a cable: each end has its IP and they talk to each other. To connect many containers, the host end is plugged into a bridge (br0, or docker0), which acts as a switch exactly like the VM bridge in 06-01. And to give outbound Internet access you add NAT with nftables, which is literally what Docker does for you.

Notice the parallel: the networking mechanism is conceptually identical to a virtual machine's (virtual interface + bridge + NAT). What changes is that here the TCP/IP stack is still the host's, only instantiated several times.

The simplest one. UTS comes from UNIX Time-Sharing System, after the structure uname() returns.

It looks cosmetic, but it is not: a great deal of software (logs, clusters, licenses, metrics) identifies itself by host name, and without this namespace every container would claim to be called the same as the host.

It isolates System V IPC objects — shared memory, message queues, semaphores — and POSIX queues, that is, much of what we studied in 03-03.

Why it matters. System V IPC identifiers are a flat, global namespace: two applications that pick the same key collide. Without this isolation, two containers running the same application would trample each other's segments. An important nuance for Meteora: the /dev/shm/meteora-cache cache is not System V, it is POSIX shared memory, which is implemented as files in /dev/shm and is therefore isolated by the mount namespace, not by ipc. They are two distinct mechanisms with the same colloquial name, and confusing them leads to real bugs.

It is the most recent of the important ones (Linux 3.8) and, without argument, the most important security improvement in the history of containers. It allows mapping ranges of UIDs and GIDs: a process can be root (UID 0) inside the namespace and correspond to an unprivileged user outside.

What just happened, which is the important part. The uid_map says: "UID 0 inside is UID 1000 outside, with a range of length 1". Inside the namespace I have all the capabilities: I can create other namespaces, mount file systems, change the host name. But when I touch a host object — /etc, which belongs to the real UID 0 — the kernel evaluates permissions with the real UID, 1000, and denies me.

The consequences are enormous. It enables rootless containers, where an ordinary user creates complete containers without sudo and without a privileged daemon, which is Podman's model. It bounds the damage of an escape: whoever breaks the isolation lands on the host as UID 1000, not as root, which turns a catastrophe into an incident. And it fulfills the least privilege of 05-01 in its purest form: the application thinks it has root and does not.

The price is a certain complexity: files created inside belong to mapped UIDs (hence newuidmap, /etc/subuid and the ranges of 65,536 UIDs per user), and some operations — mounting certain file systems, using ports below 1024 — are still not permitted.

cgroup (Linux 4.6) hides the process's real position in the control group hierarchy: inside the container, its cgroup appears to be the root, and without it a process would read the host's full path in /proc/self/cgroup, leaking the system's topology. time (Linux 5.6) allows shifting the CLOCK_MONOTONIC and CLOCK_BOOTTIME clocks; its real use case is container migration with CRIU, where on restoring on another machine the boot time must remain coherent. It does not affect the wall clock, which is still the host's.

What they do not isolate, and it is worth being very clear about: the wall clock, sysctl settings that are not namespaced, kernel modules, hardware state, dmesg and — above all — the kernel itself. A kernel failure is a failure for everyone.

Namespaces are visible in /proc/<pid>/ns/, where each one is a symbolic link whose target includes an inode number. Two processes with the same inode share that namespace.

How to read the output. Where the two inodes match, the container shares that namespace with the host; where they differ, it is isolated. On a default Docker you will see different mnt, pid, net, uts and ipc, and identical user and sometimes cgroup: that means that container does not use a user namespace, and therefore the root inside is the root outside. It is the most useful check in the whole lesson for auditing a deployment.

Two complementary commands: lsns -t pid -t net summarizes all the system's namespaces with their root process, and nsenter -t $PID -m -u -n -p -i bash enters another process's namespaces. nsenter uses setns() and is in essence what docker exec does; it is also the ultimate debugging tool, because it lets you enter the network of a container that has neither ping nor ss installed, bringing the host's along.

Namespaces control what is seen. Control groups control what is consumed. They are orthogonal mechanisms: they can be used separately, and in fact systemd has been using cgroups without namespaces for all your services for years.

Version 1 had a hierarchy per controller: one tree for cpu, another for memory, another for blkio, and a process could be in incoherent places in each of them. It was an inexhaustible source of confusion. cgroups v2 (the default in Debian 11+, Fedora 31+, RHEL 9+) imposes a unified hierarchy: a single tree mounted at /sys/fs/cgroup, where each process is in one single node and controllers are enabled per branch.

What it means. The 0:: indicates a unified hierarchy (in v1 there would be several numbered lines). The path says exactly which node of the tree your shell is in, and reflects the organization systemd imposes.

Two v2 rules cause baffling errors and you have to know them. The "no internal processes" rule says that only leaf nodes can contain processes, so moving one to an intermediate node returns EBUSY with no further explanation. And explicit delegation: for a child to be able to use a controller, the parent must enable it by writing +cpu +memory into its cgroup.subtree_control; forgetting that is the number one cause of "I wrote the limit and nothing happens".

We are going to create a group for the aggregator, which is the Meteora workload that can run away when processing the 17,280,000 bytes of the daily file, and apply real limits to it.

What just happened. A simple mkdir has created a control group with all its configuration and statistics files. It is the cgroup2 virtual file system we mentioned in the table in 04-03: the files do not exist on disk, the kernel generates them when they are read. The entire cgroup API is reading and writing text files.

What they do and how they differ, which is the most confused distinction of all. cpu.max is an absolute quota: in each 100 ms period the group can consume 50 ms, and when it exhausts them its tasks are throttled until the next period, even if the machine is completely idle; it is a ceiling. cpu.weight is the cgroup version of nice from 02-02: it only matters when there is contention and it shares out proportionally — with weights 50 and 100, one gets a third and the other two thirds if both want CPU; if one is idle, the other uses 100%.

The problem of throttling with badly set quotas deserves a paragraph of its own, because it is one of the most frequent and worst-diagnosed pathologies in the container world. Imagine meteo-api with cpu.max = 100000 100000 (one core) and an application with 4 threads. The 4 threads can run in parallel on 4 physical cores: they consume the 100 ms of quota in 25 ms of real time, and then sit completely stopped for 75 ms. The result is appalling tail latency — requests taking 80 ms when they should take 2 — with the container's average CPU at a reassuring 100%, that is, with no obvious sign of a problem.

The diagnosis is in one specific file:

If nr_throttled is a high fraction of nr_periods, you have a quota problem, not a code problem. The fixes: raise the quota, reduce the application's thread count so it matches the quota, or use cpu.weight instead of a quota if the goal was priority rather than a ceiling.

memory.max is the absolute ceiling: when the group exceeds it and nothing can be reclaimed, the OOM killer scoped to the group is invoked, killing a process from inside and not from the system. This is the crucial difference from 02-04, where the runaway aggregator could cause the death of meteo-api: with cgroups, a global failure becomes a local one. memory.high does not kill; when it is crossed, the kernel throttles the process and reclaims memory aggressively. It is a progressive brake, and good practice is to set it 20-25% below memory.max so the application has a chance to behave before dying.

Verifying the effect:

How to read it. memory.events is the best memory diagnostic in containers: high 128 says the soft threshold was crossed 128 times (there was pressure), max 47 that the ceiling was hit 47 times, and oom_kill 1 that a process had to be killed. A container that restarts with exit code 137 (128 + 9, that is SIGKILL) and has a non-zero oom_kill is dying from memory, not from an application bug.

Building on what we saw in 02-05, the io controller bounds bandwidth and IOPS per device, identified by its major:minor pair.

What it is for in Meteora. The real case is the nightly job that compresses the daily files: unlimited, it saturates the RAID 1 and meteo-api's queries suffer latencies of hundreds of milliseconds waiting for disk; with io.max the job takes longer but does not harm the service. It is the reasoning of ionice from 02-05, with a guaranteed ceiling instead of a priority.

An important nuance: io.max regulates reads and direct writes well, but those going through the page cache are accounted for when the writeback thread flushes them, not when the application performs them; for those, io.latency or the combined control with memory is more effective.

It is a one-line limit that prevents a container with a fork() loop — accidental or malicious — from exhausting the entire host's process table and making it impossible even to log in. It costs nothing to set and is one of the few limits with no downside. Always set it.

Here comes a useful revelation: you have been using cgroups for five modules without knowing it. systemd organizes the entire system into a cgroup hierarchy with three unit types: slices (system.slice, user.slice) are branches of the tree for sharing out resources, services (meteo-api.service) group a service's processes, and scopes (session-3.scope) group externally created processes.

And the limits are declared with unit directives, which systemd translates into the files we have just written by hand:

Why this matters. Remember the warning from 02-02: nice does not limit consumption. Here is the mechanism that does, and it is available without containers: adding MemoryMax and TasksMax to the hardened unit from 05-03 delivers half the benefit of containerizing with a tenth of the change. And when you run a container, Docker or Podman write into those very same files.

Namespaces and cgroups are not complete security mechanisms. A root process inside a container with no user namespace is root on the host, and it has plenty of routes to escape if it has enough capabilities. That is why every serious runtime also applies the tools from 05-01:

Docker's default seccomp profile is minimal surface well applied: it blocks mount, reboot, kexec_load, init_module, bpf and keyctl, among others, and has by itself neutralized several kernel vulnerabilities before the patch existed; disabling it "because that makes it work" is one of the worst common decisions. And no-new-privileges turns on the PR_SET_NO_NEW_PRIVS bit from the hardened unit in 05-03, so that no subsequent execve can raise privileges: it neutralizes any leftover setuid in the image in one stroke, in a single line and with no downside.

The honest conclusion: a container is secure when the three things are combined. Namespaces without trimmed capabilities and without a user namespace is toy isolation.

We are still missing the ingredient that explains why containers distribute so well. If every container needed its own complete copy of a Debian file system (about 120 MB), a hundred containers would be 12 GB of duplicated disk and a hundred different copies in the page cache.

overlayfs solves this by stacking directories. It already appeared in the table in 04-03; now we open it up. It has four pieces:

The resolution rules are simple and worth memorizing. To read, the search goes top to bottom and the first layer that has the file wins. To write a file that lives in lower, a copy-up is done: it is copied in full to upperdir and modified there, never touching the lower layer. And to delete a file from lower, since you cannot delete what is read-only, a whiteout is created in upperdir — a character device 0:0 with that name — which hides the one below.

A manual demonstration, which is the best way to understand it:

What it proves. In lowerdir=ovl/extra:ovl/base, extra is on top of base. Both have meteo-api, and extra's wins. meteora.conf is only in base and is visible all the same. That is how an image is built: every Dockerfile instruction adds a layer on top.

Now the copy-up and the whiteout:

The most important practical consequence. Copy-up copies the entire file the first time it is written, even if you change a single byte: opening a 2 GB file that comes from the image for writing copies 2 GB before your write even happens. That is why data that changes goes in a mounted volume and never in the writable layer, and why the 17.3 MB daily files in /var/lib/meteora/readings/ must not live inside the image. And it is the reason the technique works so well: a hundred containers of the same image share the same lower layers, on disk and in the page cache, so they take up 120 MB and not 12 GB. It is the linked cloning of 06-01 taken to its extreme.

With overlayfs explained, the distinction becomes trivial and stops being a source of confusion:

The binary/process analogy is exact and resolves almost every beginner's doubt: just as one /usr/bin/python3 gives rise to many independent processes, one image gives rise to many independent containers, each with its writable layer. Hence the corollary that prevents the most grief: everything a container writes in its writable layer disappears when the container is deleted. Data goes in volumes.

The word "Docker" colloquially names several layers that are worth separating, because in production they are rarely all used:

The stack goes top to bottom like this: CLI (docker, podman, nerdctl) → high-level runtime (containerd or CRI-O: images, networking, storage, lifecycle) → shim (containerd-shim, conmon) → low-level runtime (runc, crun: creates namespaces, writes cgroups, does pivot_root and executes) → Linux kernel.

runc is the reference low-level runtime, and it does exactly what we have done by hand in this lesson: it reads a JSON configuration, creates the namespaces, writes the cgroups, does pivot_root, applies capabilities and seccomp, and executes the process; its job lasts milliseconds and then it disappears (crun is a reimplementation in C, faster). containerd manages the high-level part: downloading images, mounting the overlays, configuring networks and supervising the lifecycle. Docker adds the user experience, image building and an API on top, with a daemon running as root that is its main weakness. And Podman does the same without a daemon — each container is a direct child of the user — and unprivileged, leaning on the user namespace, with a compatible CLI and systemd integration.

The OCI standards (Open Container Initiative, 2015) make all of this interchangeable: the image-spec defines the image format, the runtime-spec how a container is run from a directory and a JSON, and the distribution-spec the registry protocol. Thanks to them, an image built with Docker runs under Podman or CRI-O unchanged, and Kubernetes talks to containerd or CRI-O without needing Docker at all.

We are going to build the meteo-api image, respecting all the course's conventions: UID 990, a user with no shell, port 443, configuration in /etc/meteora/meteora.conf and data in /var/lib/meteora.

Line by line, and the reason for every decision:

• FROM python:3.12-slim: a minimal official base. The slim variant weighs ~130 MB versus ~1 GB for the full one, and every extra megabyte is attack surface and CVEs to patch. The minor version (3.12) is pinned for reproducibility; in production even the SHA-256 digest is pinned.
• Multi-stage build: the builder stage installs the dependencies, which may require a compiler and headers. The final image only copies the result. That way gcc and the development tools do not travel to production: they neither add weight nor give an attacker a way to compile an exploit inside.
• groupadd/useradd with UID 990: the same UID as on meteo-01. This is not cosmetic: when you mount the host's /var/lib/meteora as a volume, the kernel compares numbers, not names. A different UID inside and outside gives baffling Permission denied errors. --shell /usr/sbin/nologin follows the rule from 05-02 for service accounts.
• COPY --chown=990:990 and chmod 750: ownership is set at copy time, because a later chown -R over already-copied files would duplicate that layer in the overlay through the copy-up we have just studied; and the permissions are consistent with the course's umask 027.
• USER 990:990: the most important line in the whole file. Without it the process runs as root inside the container, and without a user namespace that is root on the host. With it, a compromise of the application does not even give container root.
• EXPOSE 8443: 8443 is used instead of 443 on purpose, because an unprivileged process cannot open ports below 1024 without CAP_NET_BIND_SERVICE (05-01); instead of granting the capability, it listens high and is published on 443 from outside.
• HEALTHCHECK: the runtime runs the check periodically and marks the container unhealthy if it fails three times, which is what lets an orchestrator replace it; --start-period gives startup some slack.
• ENTRYPOINT in exec form (a JSON list, not a string): in string form the process is launched under /bin/sh -c, the shell is PID 1, it does not forward SIGTERM and you are back to the signal problem.

Building and running with all the limits we have studied:

Every option and the mechanism from this lesson it corresponds to:

And the verification, which is where the circle closes with the whole lesson:

The honest comparison, without the usual exaggerations from either camp:

The practical conclusion is not "one wins", but which boundary you need. If it is a security boundary between parties that do not trust each other — different tenants, customer code, strict regulatory compliance — then a virtual machine. If what you are after is packaging, density and deployment speed within a trust domain, then a container. And the industry norm is both: containers inside virtual machines, just as we anticipated at the end of 06-01.

Calling a container a "lightweight virtual machine", or forgetting --fork with unshare --pid. The first leads to bad decisions: installing systemd and sshd inside, running several processes, treating it as a stateful server; a container is a packaged process, with no local state and replaceable. The second is the cause of the classic "I created the namespace and nothing happened": the process that calls unshare() does not enter the new PID namespace, only its children do.

Not enabling the controllers in cgroup.subtree_control. You write to memory.max and nothing happens, or the file does not even exist: the parent must delegate the controller first.

Setting a CPU quota without adjusting the threads, or storing data in the writable layer. The first is the pathology described above — with --cpus=1 and 8 threads the quota is consumed in an eighth of the period and the rest is spent throttled, with horrible tail latencies and no obvious symptoms; check nr_throttled. The second makes data vanish when the container is deleted, and it also triggers a full copy-up on every modification of a large file.

Using ENTRYPOINT in string form and building giant images. The first makes /bin/sh PID 1, which ignores SIGTERM and does not forward signals: a dirty shutdown that is always slow; use the exec form (["prog","arg"]) and --init if your process spawns children. The second — a FROM ubuntu with apt install build-essential — gives you 1.5 GB, dozens of CVEs and an arsenal of tools for an attacker: multi-stage, slim or distroless bases, and a .dockerignore so you do not pull in the whole .git.

Tip: audit with /proc/<pid>/status. The Uid, CapEff, NoNewPrivs and Seccomp fields of a container's process tell you in four lines whether the deployment is done right. It is more reliable than reading the runtime's documentation.

Tip: start with systemd, and try unprivileged first. If your problem is limiting resources and isolating a service on a specific server, MemoryMax, TasksMax and the directives from 05-03 give you almost all the benefit without changing the deployment model; containerizing makes sense when you additionally need reproducible packaging. And when you do it, start with rootless Podman or --userns-remap: if it works, you have gained the biggest security improvement available, and if it does not, the failure will tell you exactly which privilege your application is asking for and why.

Without using Docker or Podman, build a minimal "container" for the aggregator using only unshare, mount and the files in /sys/fs/cgroup. It must satisfy: (a) its own PID, mount, UTS and network namespaces; (b) its own /proc, so that ps aux shows only its processes; (c) host name aggregator-c1; (d) a limit of 256 MB of memory and half a CPU core; (e) a maximum of 50 processes. Write the commands, verify each requirement, and state what it is missing to be considered secure.

Two Meteora containers are causing trouble. For each one, say what the exact cause is, with which command you would confirm it and how you would fix it.

Container A — meteo-api, limit --cpus=1, an application with 4 worker threads. Users say that "sometimes it is dreadfully slow". The container's average CPU is 98%, memory stable at 210 MB, p50 latency 3 ms and p99 340 ms. cpu.stat: nr_periods 6000, nr_throttled 4380, throttled_usec 271000000. memory.events: all zeros.

Container B — aggregator, limit --memory=512m. It restarts every few hours with exit code 137 and no message at all in its log. It processes the 17,280,000-byte daily file by loading it entirely into memory. memory.events: high 0, max 312, oom_kill 5; memory.high is set to max.

Find all the security and efficiency problems in the following, explain the concrete risk of each one and write the corrected version.

Solution 1

Explanation. The cgroup is created first because the controllers must be enabled in the parent before the limit files exist in the child; --fork is mandatory because whoever calls unshare() does not enter the new PID namespace; --mount-proc implies --mount and remounts /proc so that ps reads the right table; and putting the PID into cgroup.procs includes the process and all its future children, which inherit the group.

Verifications:

What it is missing to be secure, and it is a lot: there is no user namespace, so the root inside is the root outside — the most serious flaw; there is no pivot_root, so it still sees the host's entire file system, /etc/shadow included; no capabilities have been trimmed, so it keeps CAP_SYS_ADMIN and CAP_SYS_MODULE; there is no seccomp, no AppArmor profile and no no-new-privileges; and /dev and /sys are the host's, with access to raw block devices. The pedagogical conclusion is the lesson's own: namespaces and cgroups on their own are not security, they are the visible half of the mechanism, and the other half is the restrictions from Module 5.

Solution 2

Container A: CPU throttling from a badly sized quota.

Cause. nr_throttled 4380 out of nr_periods 6000 means throttling in 73% of the periods, with 271 accumulated seconds of forced stoppage. With --cpus=1 the quota is 100 ms per 100 ms period, but the 4 threads run in parallel and exhaust it in ~25 ms of real time; for the remaining 75 ms all the threads are stopped, and a request arriving in that gap waits for the next period: hence a p99 of 340 ms with a p50 of 3 ms. The "98% average CPU" is misleading because it is measured against the quota, not against the machine.

Confirmation. Compare nr_throttled with nr_periods in cpu.stat: any ratio above 5% is already suspicious. And nproc inside the container will return the host's cores, which is exactly what misleads libraries that size their thread pools.

Fix, in order of preference: match the threads to the quota, because with 1 CPU of quota there is no real parallelism for 4 threads to exploit; raise the quota to --cpus=2 or 4 if the workload justifies it; or, if what was wanted was priority and not a ceiling, replace the quota with cpu.weight, which never throttles when there is free CPU. As a general measure, expose the quota to the application so it sizes its pools correctly.

Container B: cgroup OOM from a predictable memory spike.

Cause. Exit code 137 = 128 + 9 indicates SIGKILL, and oom_kill 5 confirms it was the cgroup's OOM killer, not an application failure: that is why the log is empty, the process dies with no chance to write. The max 312 says the ceiling was hit 312 times. The concrete cause is loading the 17,280,000 bytes of the daily file all at once: with Python's overhead (objects and intermediate copies) the peak is several times the raw 17 MB, and if results from several days accumulate, 512 MB is exceeded.

Confirmation. docker inspect -f '{{.State.OOMKilled}}' returns true, memory.events shows oom_kill, and journalctl -k | grep -i "memory cgroup out of memory" on the host gives the kernel trace with the chosen process.

Fix on two fronts. Immediate: set memory.high — which was at max, that is, disabled — 20-25% below the ceiling (--memory-reservation=400m) so the kernel applies pressure before killing, and raise --memory if legitimate consumption requires it, measuring the real peak first with memory.peak. Structural, which is the good fix: process the file in blocks. Since the readings are fixed-size 24-byte records, it can be read in 64 KB chunks (2,730 readings) or the file can be mapped with mmap (02-04) so the kernel manages and reclaims the pages. Consumption would go from hundreds of megabytes to a few, and the system would additionally become independent of the daily file's size.

Solution 3

Dockerfile problems:

Run command problems:

Corrected version:

With a .dockerignore that excludes .git, .env, *.conf, tests/ and __pycache__/, and with a run command exactly like the one in section 11 (--user 990:990, --read-only, --cap-drop=ALL, no-new-privileges, --pids-limit, --cpus, --memory, volumes, -p 443:8443 and --init), on tag 1.4.0 and not latest.

The key changes: the multi-stage build leaves compilers and tools out; the pinned digest makes it reproducible; the COPY of the configuration disappears and the secret comes in at run time as a read-only bind; USER 990:990 and --cap-drop=ALL implement least privilege; the port moves to 8443 and is published on 443, avoiding any network capability; ENTRYPOINT in exec form plus --init fix signals and zombies; and --privileged, -v /:/host, the daemon socket and --net=host are removed with no replacement, because there was no legitimate reason for any of the four.

The sentence that sums up the lesson is the one that opened it: a container is not a small virtual machine, it is an ordinary process of the host kernel whose view and consumption have been limited. A ps from outside proves it: the process is there, in the host's table, scheduled by the same CFS-EEVDF from 02-02 and with its pages managed by the same manager from 02-04. From that come the three consequences that govern everything: startup in milliseconds, density of hundreds per machine, and weaker isolation because the boundary is the 300-400 system calls of the shared kernel.

Namespaces are the first ingredient and control what is seen: mnt gives a file system of its own — with pivot_root as the correct mechanism and chroot as the one that was never security — pid gives a process table of its own with PID 1's two traps (adopting zombies and not ignoring SIGTERM), net gives a network stack of its own through veth pairs and a bridge, exactly like a VM, uts gives the host name, ipc isolates System V — but not /dev/shm, which is isolated by mnt — cgroup hides the position in the hierarchy, time enables migration, and user is the most important security improvement of them all, because it maps the root inside to an unprivileged user outside and turns a catastrophic escape into a bounded incident. All of it is audited in /proc/<pid>/ns/ by comparing inodes, and explored with lsns and nsenter.

cgroups v2 are the second ingredient and control what is consumed, with a unified hierarchy where the entire API is writing text files in /sys/fs/cgroup: cpu.max as an absolute ceiling — with throttling as the star pathology when threads do not match the quota, diagnosable in nr_throttled — versus cpu.weight as relative priority; memory.max with an OOM scoped to the group, which turns the global disaster of 02-04 into a local failure, and memory.high as a progressive brake; io.max so the nightly job does not choke the service; and pids.max, one line that contains any fork bomb. And the practical revelation: systemd already uses cgroups for all your services, so MemoryMax and TasksMax in the hardened unit from 05-03 deliver much of the benefit without containerizing anything.

The third ingredient is security, and without it the first two are toy isolation: trimmed capabilities, the default seccomp profile that has neutralized kernel vulnerabilities before the patch existed, AppArmor or SELinux, and no-new-privileges. overlayfs completes the picture by explaining the model's economics: a read-only lowerdir, a writable upperdir, copy-up that copies the entire file on the first change and whiteouts to delete what cannot be deleted; hence a hundred Debian containers taking up 120 MB and not 12 GB, the image being to the container what the binary is to the process, and data going in volumes and never in the writable layer. On top of it sits the real stack — runc doing in milliseconds what we did here by hand, containerd, Docker with its root daemon versus Podman with no daemon and no privileges — unified by the OCI standards.

And the hands-on part closed the circle: a multi-stage Dockerfile with a minimal base, user UID 990 so the /var/lib/meteora volume has the right permissions, port 8443 so no network capability is needed, HEALTHCHECK and ENTRYPOINT in exec form; and a run command where every option corresponds to a mechanism we studied, verifiable in four lines of /proc/<pid>/status. Compared with virtual machines, the conclusion is not that one wins, but which boundary you need: a VM to separate what does not trust each other, a container to package and densify within a trust domain, and in the industry, almost always, both at once.

That said: everything we have done has been on one machine. We have limited, isolated and packaged meteo-api, but there is still a specific meteo-01, with a specific disk, that somebody installed and that leaves the service down if it breaks. Who decides on which machine each container runs when there are thirty machines? What happens when one dies at three in the morning? How do you share a data file between containers that are not even on the same server? And what is left of the operating system when the machine stops being a physical object and becomes an API call that returns an instance in forty seconds — and that can disappear with the same amount of notice?

That is where the requests and limits we have just written by hand in /sys/fs/cgroup become declarations in a YAML file, and the scheduler from 02-02 reappears one level up, distributing containers across nodes instead of processes across cores. That is The Operating System in the Cloud.