In lesson 03-06 you learned to use volumes, bind mounts and tmpfs. Here you will see what lies underneath: how the daemon stacks an image's layers with overlay2, how much writing to a container's layer really costs, what volume drivers are, and how to set up a backup strategy for Aurora Libros with tested restores.

1. What a storage driver is and what it solves
2. overlay2 from the inside: the four directories
3. Real inspection of an image and a container
4. Comparison table of the storage drivers
5. Checking the active driver and changing it
6. The cost of copy-on-write, measured
7. Why databases demand volumes
8. The local driver and its driver_opts
9. Mounting NFS and tmpfs as a volume
10. Third-party plugins and declaring them in Compose
11. external volumes shared between stacks
12. The Aurora Libros backup strategy
13. A backup script with rotation and verified restore
14. Disk space and data-root
15. Performance and permissions (UID/GID)

An image is a stack of read-only layers (lesson 01-05). A container adds a writable layer on top. The technical problem is this: presenting all those layers as a single coherent filesystem, and doing it without copying gigabytes every time a container starts.

That is the job of the daemon's storage driver. It has three responsibilities:

• Stacking the image's layers into a single directory tree visible from inside.
• Applying copy-on-write: when a file from a lower layer is modified, copy it to the upper layer and work on the copy.
• Sharing common layers between containers, so that ten instances of node:22-alpine take up the space of one.

It is a piece of the daemon, not of the container: from inside all you see is an ordinary filesystem.

overlay2 uses OverlayFS, a union filesystem included in the Linux kernel. It works with four concepts:

The resolution rules are simple, and they explain all the behavior you have already observed:

1. Reading a file: it is looked up in upperdir; if it is not there, lowerdir is walked from top to bottom. The first match wins.
2. Writing to a file that only exists in lowerdir: it is copied whole to upperdir (copy-up) and written there. The original is untouched.
3. Deleting a file from lowerdir: you cannot delete something that is read-only, so a special whiteout file is created in upperdir to hide it. The original still takes up space.

That third rule is, at last, the explanation for why deleting files in a later Dockerfile layer does not reduce the image's size. You will see it measured in lesson 05-04.

All of this is visible on the host's disk. Each layer lives in /var/lib/docker/overlay2/<id>/:

Notice the short links under overlay2/l/: they exist because the length of the kernel's mount arguments is limited, and an image with many layers would exceed the limit if full paths were used.

Now the demonstration that ties everything together. Write a file inside the container and watch it appear in the host's upperdir:

The file is physically on the host, inside that container's diff and that container's alone. It is the same thing docker diff showed you in lesson 03-03, except now you know where it lives.

Practical rule: if docker info says overlay2, do not touch anything; if it says vfs on a server, you have a configuration problem that will multiply your disk usage by ten.

You change the driver in /etc/docker/daemon.json with {"storage-driver": "overlay2"} and restart the daemon.

Warning. Changing the storage driver makes the daemon stop seeing all existing images and containers: they are still on disk, under the old driver's directory, but they are inaccessible until you go back. Before touching it, push your images to a registry, save any unpublished ones with docker save, and back up your volumes. Volumes, on the other hand, do not depend on the driver: they live in /var/lib/docker/volumes/ and survive the change.

Theory says copy-up copies the whole file the first time you touch it. Let's measure it with a 512 MB file baked into an image.

Almost two seconds to write one byte, and 512 MB of writable layer consumed.

That is the real cost of copy-on-write, and it explains three rules you had already been applying without knowing why:

PostgreSQL writes non-stop to data files of tens or hundreds of megabytes. If /var/lib/postgresql/data were in the writable layer, every page modification would trigger a copy-up of the entire file the first time, and performance would be catastrophic.

A volume bypasses OverlayFS completely: it is a bind mount to a host directory (/var/lib/docker/volumes/<name>/_data), mounted over the corresponding path. Writes go straight to the host's filesystem, with no layers and no copy-up.

There is the proof: while / is overlay, /var/lib/postgresql/data is the host's filesystem directly. That is why official database images declare that path as a VOLUME: if you forget to mount one, they at least avoid the penalty by creating an anonymous volume.

The default volume driver is called local, and it accepts options almost nobody uses, even though they are surprisingly powerful: they are the same ones as Linux's mount command.

The difference from a plain bind mount is one of management: a named volume shows up in docker volume ls, accepts labels, can be declared external, and does not depend on the project's relative path.

An NFS volume lets several hosts share the same data, something impossible with a local volume:

Three warnings from the field. The mount happens when the container starts: if the NFS server does not answer, the container does not start. Use soft and a reasonable timeo so that a network failure produces an error rather than a process stuck forever in state D. And never put PostgreSQL's data on NFS: file locking over the network does not offer the guarantees a database needs, and corruption is only a matter of time. For static files —the catalog's cover images— it is perfect.

A volume driver is a daemon plugin that services mount requests. Docker ships local; everything else is installed:

In Compose they are declared like any other volume, with driver: rclone and its driver_opts (remote: "s3aurora:backups"). The important warning: a volume plugin is privileged software inside your daemon. Install only plugins from verified sources, and treat them with the same scrutiny you would apply to approving a production dependency.

Compose prefixes the project name to the volumes it creates (aurora-libros_aurora-data). When a volume has to exist outside the project's lifecycle —or be shared with another stack— you declare it external:

Two very practical consequences. First: docker compose down -v does not delete an external volume, which makes it a cheap safety net for the data you cannot afford to lose. Second: another stack —a separate reporting project, for instance— can mount the same volume as :ro and read the data without touching the main platform.

There are two ways to back up aurora-db's data, and they are not interchangeable:

Logical backup, taken hot:

The -T is mandatory: without it, exec allocates a pseudo-terminal and corrupts the binary stream with line-ending translations. It is the classic mistake in this operation, and it gives no warning at all: the file looks fine and fails on restore.

File-level backup of the volume:

Why the service has to be stopped: PostgreSQL keeps modified pages in memory and writes asynchronously. A tar over a live data directory captures some files at one instant and others at another, producing an incoherent set that can restore without errors and then corrupt itself days later. With the service stopped, the directory is still and the backup is valid.

What makes this script serious is not the backup, it is the verification: every run restores the file it has just created into an ephemeral PostgreSQL on tmpfs and counts the rows. If the backup was empty, truncated or corrupt, the script fails today, not on the day of the disaster.

Burn the rule into your memory: a backup that has never been restored is not a backup, it is a folder with hope inside.

Everything Docker stores lives under /var/lib/docker:

If containers/ is enormous, the culprit is unrotated logs (you will fix that in 05-06). If overlay2/ is, it is old images. To move the store to a bigger disk:

Use rsync -aHAX and not cp: hard links, extended attributes and permissions have to be preserved, or the layers will be useless. Check that everything works before deleting the old directory.

Choosing a mount by performance, from best to worst for write-intensive workloads:

Permissions are the other half of the problem. An empty volume inherits the owner of the image's mount point; a volume with data keeps its own, and aurora-api runs as the node user (UID 1000), so a volume created by root is unreadable to it.

With bind mounts the fix is different, because the owner is set by the host: either you align the container's UID with your user's (user: "${UID}:${GID}" in Compose), or you adjust permissions on the host. And remember from lesson 03-06 that the kernel only understands numbers: the name node means nothing outside the container.

Writing persistent data to the container's layer. Terrible performance because of copy-up, and total loss when the container is recreated.

Running tar on a volume while the database is up. The backup looks fine and is incoherent. Stop the service or use pg_dump.

Forgetting -T in docker compose exec pg_dump. The pseudo-terminal corrupts the binary dump without warning.

Never testing a restore. On the day of the incident you find out you have been storing empty files for months.

Running docker volume prune without looking. It deletes every volume not used by any container, and a volume belonging to a stopped stack fits that definition. Declare anything critical as external.

Changing the storage driver without a prior backup, or putting PostgreSQL data on NFS. In the first case images and containers vanish from the inventory; in the second, network file locking does not give the guarantees a database needs.

Tip: label your volumes when you create them (--label criticality=high) and filter by that label before any cleanup: docker volume ls --filter label=criticality=high. A minute of labeling avoids the deletion that has no undo.

Exercise 1. Demonstrate copy-on-write in both directions: locate aurora-api's upperdir, create a file inside the container and find it on the host's disk; then measure the cost of modifying one byte of a 256 MB file that comes in the image, and compare it with modifying one byte of a file already present in the writable layer.

Exercise 2. Set up the full backup and restore cycle for aurora-db: take a logical backup, destroy the data by deleting the volume, restore, and check that the nine titles are back. Explain at exactly which point the platform would have been left unusable without the backup.

Exercise 3. Turn aurora-data into an external volume, prove that docker compose down -v no longer deletes it, and explain what protection this gives and what its downside is.

Solution 1.

One second versus zero, to write exactly the same byte. The difference is the copy-up: in the first case the file lives in a read-only lowerdir and the kernel has to copy all 256 MB to the upperdir before allowing the write; in the second, cp had already brought it into the writable layer. Extrapolate: PostgreSQL writing non-stop to data files without a volume would pay that toll over and over, on top of multiplying the space it occupies. Here you can see, in one second of wall clock, why the decision made in lesson 03-06 was not a convention but a physical necessity.

Solution 2.

There is a nuance here that confuses a lot of people: after the down -v the database comes back with 9 books without anything having been restored, because the volume has been recreated empty and PostgreSQL has run db/init.sql again. That is exactly what makes the situation dangerous: it looks as if nothing was lost.

What has been lost is the data added after initialization, which is everything that matters in production:

After the disaster 9 remain (the ones from init.sql); after the restore, the real 10. The exact point at which the platform would have been left unusable is the down -v: the -v deletes the volumes, and with them the only place where the user-added data existed. Without /tmp/aurora10.dump, that row is not recoverable from anywhere.

Solution 3.

The down -v has not deleted the volume and the data is still intact: Compose refuses to destroy what it did not create. The protection is real and costs two lines of YAML, so for the data volume of a production platform it is an almost automatic decision.

The downside has two faces. The first is operational: the volume no longer creates itself, and whoever deploys for the first time on a new server has to run docker volume create before the up, or Compose will fail with external volume not found. The second is conceptual: you lose the reproducibility of an environment built from scratch. In development that is annoying —the reset from lesson 04-07 stops working— so the sensible practice is to declare external only in compose.prod.yaml, leaving the local environment disposable.

You now know what happens beneath -v. A storage driver stacks an image's layers and applies copy-on-write, and overlay2 does it with four directories you have seen on disk: the image's read-only lowerdirs, the upperdir where every file the container writes appears, the merged view that the process sees as /, and the kernel's internal workdir. From those come the three resolution rules, including the whiteout rule that explains why deleting a file in a later layer frees no space. You know the full driver table —btrfs, zfs, the retired devicemapper, fuse-overlayfs for rootless and the painfully slow vfs— and you know that changing driver makes all your images invisible.

And you have measured copy-on-write: almost two seconds and 512 MB of layer to write a single byte to a file that came in the image. That number turns a recommendation into physics: a database writes constantly to large files, therefore it goes in a volume, which bypasses OverlayFS and writes straight to the host's filesystem. You know how to squeeze the local driver with its driver_opts —a named bind, a sized tmpfs, NFS for the cover images and never for PostgreSQL—, how to declare third-party plugins in Compose, and how to use external volumes so that down -v cannot take production data with it. You also take away a backup strategy that is a strategy and not an intention: the table separating the hot logical backup with pg_dump from the file-level backup with tar —and why the second requires stopping the service—, a script with rotation that restores and counts rows on every run, the rule that a backup which has never been restored is not a backup, and control over the space in /var/lib/docker through data-root.

Lesson 05-03 brings security, and with it the full hardening of Aurora Libros: the threat model and its four surfaces, why belonging to the docker group is equivalent to being root —with the escalation demonstrated—, rootless mode, minimal images pinned by digest, --read-only, kernel capabilities, seccomp, vulnerability scanning with Docker Scout and Trivy, SBOM and signing with Cosign, and a hardened compose.prod.yaml line by line with its actionable checklist.