We closed module 4 with a platform that was reachable, encrypted and segmented, and with a debt written into a manifest comment back in module 2: bookings-postgres has no persistent storage. This module settles it, and it starts at the beginning, which is not the PersistentVolume but something more basic: understanding why a container's filesystem disappears and what exactly a volume is in Kubernetes. You will see that the volume is a property of the pod, not of the container, and that this distinction solves a container restart but not a pod replacement — precisely the problem the database has been dragging along. In this lesson you will demonstrate the data loss with your own hands, master the volumes/volumeMounts pair that governs every volume type in the rest of the module, and work in depth on the two types that depend on no external infrastructure: emptyDir and hostPath.

1. Why a container's filesystem is ephemeral
2. Demonstration: losing data in bookings-postgres
3. What a volume is in Kubernetes
4. The volumes and volumeMounts pair
5. mountPath, readOnly, subPath and subPathExpr
6. emptyDir in detail
7. Sharing an emptyDir between containers of the same pod
8. hostPath: what it is and why it is dangerous
9. The configuration volumes: configMap, secret, downwardAPI
10. The projected volume: everything in one directory
11. Summary table of volume types
12. Generic ephemeral volumes: the bridge to the rest of the module

A container image is made up of read-only layers stacked on top of each other. When the runtime (containerd in your minikube) starts a container from postgres:16.4, it does not copy those layers: it mounts them stacked by means of a union filesystem (overlayfs) and adds a single writable layer on top, empty and exclusive to that container.

Everything the process writes — a new file, a change to an existing one, PostgreSQL's data files under /var/lib/postgresql/data — goes to that writable layer. And that layer has a devastating property: its lifecycle is exactly the container's. When the container is destroyed, the layer is deleted.

It is worth telling apart, with precision, two events that sound alike and are not:

In other words: any restart, of whatever kind, wipes out what was written. And this is not a defect of Kubernetes; it is the very definition of a container and the reason they are lightweight and replaceable. The price is that state has to be taken out of the container explicitly, and that "outside" is the volume.

You will remember from 03-04 that this writable layer consumes the node's ephemeral-storage and that a container filling it up causes the pod to be evicted. Volumes are part of that accounting too, as we will see with emptyDir.

Do it. It is the exercise that gives the whole module its meaning. We start from the current state: bookings-postgres running as a Deployment in rutas-norte-dev, with no volume at all. We create a real booking inside the database and, on top of that, an arbitrary witness file:

Now we trigger what in production would be caused by a deployment, an eviction or a node failure:

Everything is gone. The ReplicaSet did its job perfectly — there is a Running pod, the Service routes, the application starts — and yet the business has lost every booking. This is the exact point that separates a stateless workload from a stateful one: for web-store, recreating the pod is free; for bookings-postgres it is a catastrophe.

A volume is a directory, whose content and backing vary with its type, that Kubernetes makes available to the containers of a pod.

The formal definition matters less than these two statements, which are the ones to internalise:

1. The volume is declared on the pod, not on the container. It lives in the pod's spec.volumes. Containers merely mount it at a path, through spec.containers[].volumeMounts. That is why two containers in the same pod can see the same volume at different paths.
2. The volume's lifecycle is tied to the pod, not to the container. A volume survives a container restart. What happens when the pod disappears depends on the volume type: some die with it (emptyDir) and others do not (those backed by a PersistentVolumeClaim).

That second statement explains exactly how far this lesson goes. An emptyDir fixes the case "the PostgreSQL container crashed and the kubelet restarted it": the new container mounts the same directory again and finds its data. It does not fix the case "the pod was replaced", which is the one you have just demonstrated. For that you need the PersistentVolumes of 05-02.

Every volume requires two declarations linked by name. It is the most repeated pattern in Kubernetes and it does not change whatever the type:

The two blocks, precisely:

• spec.volumes (pod level): a list of volumes. Each element has a name that is unique within the pod and exactly one type key (emptyDir, hostPath, configMap, secret, persistentVolumeClaim, projected, ephemeral...). Declaring a volume that no container mounts is legal but useless; for some types (such as configMap) it still means the pod will not start if the referenced object does not exist.
• spec.containers[].volumeMounts (container level): where that volume is seen inside that particular container. The name is the cross-reference.

If the name does not match, the error is immediate and explicit when the pod is created:

initContainers can mount volumes too, and that is precisely the mechanism that makes the initialisation pattern studied in 06-04 useful: an init container prepares files in a volume and the main container finds them ready.

The volumeMounts fields deserve detail because each one solves a real problem.

This is the effect that surprises people most at first. If you mount a volume at /etc/nginx/conf.d in the web-store image, everything the image shipped in that directory stops being visible, just as when you mount a disk over a directory in Linux. It is not deleted: it is covered up. That is why mounting over /etc or /usr breaks the container.

You already used it in 03-01 to inject a single configuration file without covering the whole directory. The same technique works for data volumes:

In PostgreSQL's case there is a specific operational reason to do this, explained in detail in 05-03: many formatted block volumes come with a lost+found directory at their root, and PostgreSQL refuses to initialise a data directory that is not empty. In Rutas Norte we already solved it back in module 3 through the equivalent route of the PGDATA variable, and in 05-03 we will bring both approaches together.

subPathExpr expands the container's environment variables, including those from the Downward API of 03-03. The canonical case is having each replica write into its own subdirectory of a shared volume:

Each notifications-worker pod writes its logs into a subdirectory named after itself, and they never tread on each other.

emptyDir is the simplest volume type and by far the most used.

When it is created: at the moment the pod is assigned to a node. It always starts empty, hence the name.

When it is deleted: when the pod is removed from the node, definitively and irrecoverably. Let it be crystal clear what survives it and what does not:

Where it lives: by default, on the node's disk, under /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/. You can see it in minikube:

With no limit, an emptyDir can fill the node's disk and cause DiskPressure, with the eviction of unrelated pods that you studied in 03-04. It is capped with sizeLimit:

When the limit is exceeded, the kubelet evicts the pod (it does not return a write error to the process). The event says so bluntly:

With medium: Memory, the volume is backed by a tmpfs: RAM, not disk. It is blazingly fast and the content never touches a disk, which makes it ideal for sensitive material — it is exactly what Kubernetes does under the bonnet with secret volumes, as you saw in 03-02. It is declared with emptyDir: { medium: Memory, sizeLimit: 64Mi }.

The important warning: whatever you write there counts against the memory limit of the container using it. If web-store has limits.memory: 256Mi and you fill 200 MiB of tmpfs, the process is left with 56 MiB and will end up OOMKilled without any "process memory" graph explaining it. Rule: if you use medium: Memory, add it to the memory limit and always set sizeLimit. Without sizeLimit, the tmpfs can grow to half the node's RAM.

This is the use case that on its own justifies the existence of emptyDir, and it fits what you learned in 02-01: the containers of a pod share the network and can share volumes, but not their filesystem.

In Rutas Norte, web-store renders HTML templates of timetables and routes. We add a secondary container that generates them every five minutes from the API and leaves them in a directory; nginx simply serves them. Both see the same emptyDir, and nginx mounts it read-only.

The design points to read out of that manifest:

• One single volume, two different mountPath values. The generator writes to /output; nginx reads from /usr/share/nginx/html/timetables. It is the same directory on the node.
• readOnly: true on the consumer. If tomorrow a vulnerability allows writing through nginx, the attacker cannot alter the templates it serves. It costs one line.
• medium: Memory with sizeLimit: 64Mi, and nginx's limits.memory raised to 192Mi to absorb it. Without that adjustment, a spike in templates would kill the container with OOMKilled.
• This is not persistence. If the pod dies, the cache is lost and the generator rebuilds it on the next start-up. That is exactly the criterion for choosing emptyDir: rebuildable data.

Verification, including proof that readOnly works:

hostPath mounts a file or directory from the node's filesystem inside the pod.

The type field validates what is expected to be found at that path, and it is the difference between a clear failure and silent, unexpected behaviour:

Legitimate uses, all of them infrastructure components, not applications:

• A log collection agent that reads /var/log/pods on every node (the DaemonSet pattern of 06-02 and of the EFK stack of 07-05).
• A node metrics exporter that reads /proc and /sys.
• A CSI plugin that needs access to the kubelet's mount directory.

hostPath breaks the isolation between the pod and the node. The consequences are of the highest severity:

1. Escalation to root on the node. A pod mounting / or /etc with write access can add an SSH key to /root/.ssh/authorized_keys, or drop a manifest into /etc/kubernetes/manifests/ that the kubelet will start as a static pod with full privileges. That is total control of the node.
2. Access to other pods' secrets. Under /var/lib/kubelet/pods/ are mounted the volumes of every pod on that node, including the Secrets with the credentials of bookings-postgres.
3. Runtime escape. Mounting /var/run/containerd/containerd.sock (or the Docker socket) allows privileged containers to be created without going through the Kubernetes API or RBAC.
4. Data that does not travel. Even if there were no security risk, a hostPath is tied to one specific node: if the pod is rescheduled elsewhere, it finds a different (or empty) directory and the data "vanishes" with no error at all.

That is why hostPath is not a persistence solution for bookings-postgres, even though at first glance it looks like one. And that is why the Pod Security Standards at their baseline level forbid hostPath outright; you will see it enforced in 08-03, and the accompanying container hardening in 08-02.

Rutas Norte rule: no application manifest uses hostPath. If one shows up in a code review, it is rejected without discussion. The legitimate need — real persistence — is covered with a PVC.

You have already used all three in module 3; we put them back here because they are volumes, with the same volumes/volumeMounts mechanics, and it is worth seeing them as one family.

Two reminders that are often forgotten:

• Automatic content updates do not restart the process. That is why in 03-03 we added an annotation with the configuration hash to the pod, to force a rollout when it changes.
• A mount with subPath is frozen: it receives the content as of the moment of mounting and is never updated. It is the trade-off for being able to mount a single file.

projected combines several sources — configMap, secret, downwardAPI and serviceAccountToken — into a single directory. It serves to avoid filling the container with mount points and, above all, to request ServiceAccount tokens with a controlled audience and expiry, as you saw in 03-06.

Mounted at /etc/rutasnorte, the container sees a single tree: app/application.yaml, db/password and token.

Restrictions to remember: inside a projected there is no room for persistentVolumeClaim or emptyDir (only the four sources listed), and the same path cannot be repeated in two sources: the pod is rejected for conflict.

The types in use today on a 1.30+ cluster, with their lifecycle and their use case. The "in-tree" types of specific providers (awsElasticBlockStore, gcePersistentDisk, azureDisk...) are removed or migrated to CSI; do not write them in new manifests.

One type remains that acts as a hinge between this lesson and the next ones: the generic ephemeral volume. It is a volume that is provisioned like a PVC — with all the power of real storage: whatever class you want, whatever size you want, expansion, snapshots — but whose lifecycle is the pod's: when the pod is deleted, the automatically created PVC is deleted with it.

This is what to use when you need a lot of temporary space — more than reasonably fits on the node's disk — without wanting to keep it: generating a bulky report, batch processing, a large download. In Rutas Norte it will fit occupancy-reports when it arrives in 06-03.

The comparison closes the lesson's mental map:

And with that, what is missing is laid out: for bookings-postgres we need the third column. That column is built by the five remaining lessons of the module.

Four tips that save hours:

1. Before writing a volume, answer this: what happens if this is lost? If the answer is "it gets regenerated", emptyDir. If it is "we lose business data", a PVC. There is no middle ground.
2. Inspect the effective volumes of a running pod, which are not always the ones you thought, with kubectl exec ... -- df -h and kubectl exec ... -- mount | grep -E "tmpfs|overlay".
3. kubectl describe pod shows the Volumes: and Mounts: sections for each container: it is the quickest place to check that the mount is the one you intended and whether it is ro or rw.
4. kubectl explain pod.spec.volumes lists every type available in your cluster version, which is the source of truth against any out-of-date documentation.

In rutas-norte-dev, create a pod ephemeral-test with the image busybox:1.36 and two volumes: a normal emptyDir mounted at /survives-restart and another emptyDir with medium: Memory and sizeLimit: 32Mi mounted at /in-ram. Write a file in each one. Then:

1. Kill the container's main process to force a restart and check what survives.
2. Delete the pod, recreate it and check what survives.
3. Use mount to verify which of the two is on tmpfs.
4. Try to write 50 MiB into /in-ram and describe what happens.

Write k8s/base/web-store-cache.yaml: a web-store Deployment in rutas-norte-dev with two containers sharing an emptyDir called template-cache:

• nginx (image nginx:1.27.1) mounting it at /usr/share/nginx/html/timetables read-only.
• template-generator (use busybox:1.36 with a loop that writes the time to an HTML file every 30 seconds) mounting it at /output read-write.

Labels and resources following the course conventions. Show that nginx sees what the generator writes and that it cannot write itself.

A colleague submits this manifest to give persistence to bookings-postgres in the pre-production cluster. Find four distinct problems and explain the consequence of each.

Exercise 1

Both survive: the volume is tied to the pod, not to the container. Even the RAM one, because the tmpfs belongs to the pod.

Both directories are empty: it is exactly what happens to bookings-postgres today. And with medium: Memory the sizeLimit is the real size of the tmpfs, so the failure is immediate and inside the container. On a disk-backed emptyDir the behaviour is different: the write succeeds and it is the kubelet that, on detecting the excess, evicts the pod with the event Usage of EmptyDir volume ... exceeds the limit. Check it with kubectl get events -n rutas-norte-dev --sort-by=.lastTimestamp.

Exercise 2

Exercise 3

The four problems:

The fix is not to touch up this manifest but to change mechanism: a PersistentVolumeClaim, which is what the following lessons build.

You have demonstrated with your own hands the problem that gives the module its name: you created a booking in bookings-postgres, deleted the pod and the booking disappeared. The cause is not a bug: it is the overlayfs writable layer, which is born empty with each container and dies with it. From there comes the rule that governs everything else: in Kubernetes, state has to be taken out of the container explicitly.

That "outside" is the volume, and you can now read it precisely. It is declared on the pod (spec.volumes), not on the container; containers only mount it (volumeMounts), and both blocks are linked by the name. You have mastered the mount fields: mountPath, which hides whatever was at that path; readOnly, which you should set by default on every consumer; subPath, to mount a single file at the price of freezing its content; and subPathExpr, to give each replica its own subdirectory.

You know in depth the two types that depend on no external infrastructure. emptyDir is born empty when the pod is assigned to a node and dies with the pod: it survives a container restart but not its replacement; it is capped with sizeLimit so as not to cause DiskPressure, and with medium: Memory it lives in RAM — fast and never touching disk, but counting against the container's memory limit. You have put it to work in the case it belongs to: the template cache shared by nginx and the generator inside the web-store pod, with the consumer read-only and the data rebuildable. And hostPath, with its type values and its clear verdict: legitimate only for infrastructure, forbidden in applications, because it breaks isolation with the node — root escalation, access to other pods' secrets, escape through the runtime socket — and because it ties the data to one specific machine. You have also put back in place the configuration volumes you were already using — configMap, secret, downwardAPI — and the projected one that gathers them into a single directory, and you have the table of every type with its lifecycle.

What you have not solved is precisely what opened the lesson: none of the volumes seen survives the replacement of the pod. We left the hinge marked with the generic ephemeral volume (ephemeral.volumeClaimTemplate), which is already provisioned as real storage even though it still dies with the pod. One last thread remains to be cut: decoupling the life of the disk from the life of the pod. That calls for a new object, with its own existence in the cluster, administered by whoever manages the infrastructure and consumed by whoever deploys the application. It is the PersistentVolume, and it is the next lesson: Persistent Volumes.