The previous lesson ended with a PersistentVolume that appeared on its own, and with a black box left unopened: we said "the provisioner creates the volume" without explaining who that provisioner is, how it finds out there is a pending PVC, how it attaches the disk to a node and how it ends up mounted in the container. That machinery is called CSI, the Container Storage Interface, and understanding it is what separates someone who applies manifests from someone who can diagnose why a pod has spent twenty minutes in ContainerCreating. Besides, the StorageClass announced two capabilities we have not used yet: expansion of a running volume and snapshots, indispensable before touching the schema of a production database. In this lesson you will put them to work on bookings-postgres, and you will close with the warning that prevents the most incidents: a snapshot is not a backup.

1. What problem CSI solved
2. The architecture: controller plugin and node plugin
3. The sidecars and the CSIDriver and CSINode objects
4. The complete end-to-end flow of a volume
5. Preparing the practice cluster
6. Volume expansion: requirements and mechanics
7. Online expansion versus expansion with a restart
8. Snapshots: the three objects
9. Creating a snapshot of bookings-postgres
10. Restoring from a snapshot with dataSource
11. Cloning a PVC
12. The warning: consistency and what a snapshot is NOT

Before 2018, the code that talked to AWS EBS, to Google Persistent Disk, to Ceph or to NetApp lived inside the Kubernetes repository. They were called in-tree drivers, and the model had three serious problems:

CSI (Container Storage Interface) is a standard specification — not exclusive to Kubernetes: Mesos and Nomad use it too — defining a gRPC contract between the orchestrator and the storage system. The provider implements that contract in a component of its own, packages it as a container and deploys it in the cluster, like any other workload. The result: the in-tree drivers became obsolete and were removed (kubernetes.io/aws-ebs, kubernetes.io/gce-pd, kubernetes.io/azure-disk no longer exist in 1.30+). Today all network storage goes through CSI, and there are more than a hundred drivers available.

A CSI driver is always deployed in two halves, because storage operations are of two different natures:

The separation is not arbitrary: creating a disk is a call to a remote API that should only be made once from one place; mounting it is a local operation that only the machine where the pod is can perform.

Here is the elegant part of the design. The provider's driver does not talk to the Kubernetes API: it only implements the CSI gRPC contract. What watches the API and translates are some helper containers maintained by the Kubernetes project, the sidecars, deployed alongside the driver in the same pod (the multi-container pattern of 06-04).

There is also a livenessprobe that probes the driver's health. Thanks to this design, a vendor only writes the logic of its storage; all the Kubernetes integration is already written and tested. And there are two API objects that describe the state of this infrastructure:

It declares the capabilities of an installed driver. The cluster consults it to know what it can ask of it.

It is created automatically for each node and records which drivers are available there and by what identifier the provider knows that machine.

That Count is a very real limitation in production: AWS, for instance, restricts the number of EBS volumes that can be attached to an instance. When it runs out, pods stay Pending with node(s) exceed max volume count, a message you now know the origin of.

This is the diagram to remember: what happens exactly from the moment you apply a PVC until the process writes its first byte.

The five phases, with their technical names and the symptom when they fail:

Two details worth their weight in gold when diagnosing. NodeStageVolume happens once per node; NodePublishVolume, once per pod, which is why a volume can be "mounted" and yet the pod not see it: they are two different mounts. And you can follow the attach phase in the API with kubectl get volumeattachments, something rarely known: it shows one row per attached volume, with the driver, the PV, the node and an ATTACHED column; if that column is false for minutes, the problem is in the controller plugin, not on the node.

The minikube provisioner (k8s.io/minikube-hostpath) supports neither expansion nor snapshots, so for this lesson we will install the hostpath CSI driver, which does implement them. It is a reference driver for testing — do not use it in production — but it faithfully reproduces the behaviour of a real CSI.

Important point: the snapshot CRDs and the snapshot-controller are not part of Kubernetes. On a managed cluster they usually come installed, but on your own you have to add them. If you try to create a VolumeSnapshot without them, the error is no matches for kind "VolumeSnapshot" in version "snapshot.storage.k8s.io/v1".

Now we rewrite rutasnorte-fast on top of this driver — it has to be deleted and recreated, because StorageClasses cannot be edited —, keeping the contract (the name and Retain) and gaining the two capabilities:

The day comes when 10 GiB is not enough: the bookings table takes up 8.4 GiB and the disk is on course to fill up in two weeks. Expansion lets you grow the volume without migrating the data.

The requirements, all three at once: the StorageClass must have allowVolumeExpansion: true, the CSI driver must implement the capability (ControllerExpandVolume) and the PVC must be Bound.

And the absolute restriction: a volume cannot be shrunk. Not with a new PVC, not with kubectl edit, not in any way; Kubernetes rejects the request with spec.resources.requests.storage: Forbidden: field can not be less than previous value.

The reason is data safety: shrinking a filesystem requires moving and compacting it, and no provider guarantees doing so without risk. Practical consequence: ask sensibly, because you can only go up. Shrinking means creating a smaller PVC and copying, with downtime, as in 05-04.

You edit the PVC's spec.resources.requests.storage — never the PV's. The quick route is a patch; the proper one is editing the manifest in Git and applying it, as the declarative model of 01-06 requires:

And the check that really matters, inside the container:

20 GiB, with the 8.4 GiB of data intact and without having stopped the database.

Expansion happens in two stages, hence the two conditions you will see on the PVC:

If the PVC is left with the FileSystemResizePending condition, the disk is already bigger but the filesystem does not know it: df -h inside the container still shows the old size. The fix is to recreate the pod, and the kubelet runs NodeExpandVolume during the mount:

Two operational warnings. First, subPath can prevent the automatic expansion of the filesystem on some drivers: it is the reason, flagged in 05-03, why on bookings-postgres we prefer the PGDATA variable to subPath. And second, watch how full it is getting before it becomes urgent: Kubernetes exposes kubelet_volume_stats_available_bytes and kubelet_volume_stats_capacity_bytes, and an alert at 75% gives you plenty of room (07-03 and 07-04).

A snapshot is a point-in-time copy of a volume's content, taken by the storage system. Its great virtue is that it is almost instantaneous — modern systems use copy-on-write, so they do not duplicate the data on creating it — and that it allows new volumes to be created from it. The object model is an exact replica of the volumes one, which means you already know it:

And the VolumeSnapshotClass, analogous to the StorageClass:

deletionPolicy decides what happens to the real snapshot when the VolumeSnapshot object is deleted: Delete destroys it; Retain keeps it, and it is the choice consistent with rutasnorte-fast for the database snapshots.

Prerequisites, which you already covered in section 5: the three CRDs, the snapshot-controller and the external-snapshotter sidecar alongside the driver.

The real scenario: tomorrow version 3.0 of bookings-api is deployed, and it includes a schema migration — it adds columns to bookings and rewrites the customers table. If the migration goes wrong, you have to go back in minutes, and a rollout undo of the Deployment (02-04) does not undo the changes in the database. A prior snapshot does.

Starting state: three bookings in the table, according to kubectl exec ... psql -c "SELECT count(*) FROM bookings;". The snapshot manifest:

READYTOUSE: true is the column to look at: until it is, the snapshot is no use for restoring. The describe adds the Bound Volume Snapshot Content Name, the Creation Time, the Restore Size and the CreatingSnapshot and SnapshotCreated events from the snapshot-controller.

The annotations with the reason, the owner and the ticket are not decoration: a cluster accumulates snapshots, each one costs money, and without that information nobody dares delete any of them. It is the direct application of what you learned in 02-07.

Now we simulate the disaster. The migration runs and goes badly: it deletes bookings.

Restoring does not overwrite the original volume: it creates a new PVC whose dataSource points at the snapshot. It is a crucial difference, because it preserves the evidence of the damaged state so it can be analysed afterwards.

Applying it brings up a second Bound PVC alongside the original, with its own PV. And we point the Deployment at the restored volume:

The three bookings are back. Four conditions for a restore to work:

1. The new PVC must ask for at least the snapshot's restoreSize. Less, and it fails.
2. It must use the same StorageClass (or at least the same driver): a snapshot does not cross storage systems.
3. The snapshot must be READYTOUSE: true.
4. The snapshot must exist in the same namespace as the new PVC. To cross namespaces you have to import the VolumeSnapshotContent by hand.

A sibling case and a very useful one: creating a copy of a volume straight from another PVC, without going through a snapshot. Only the dataSource's kind changes:

Cloning is the clean way of giving the development team a copy of the pre-production data to rehearse a migration. And mind the small print: that clone contains real personal customer data — name, ID number, phone number, email address. A clone for testing is data processing exactly as the original is and demands the same protections, or better still, prior anonymisation. It is picked up again in 05-06.

One technical limitation worth knowing: cloning a PVC while the database is writing produces a copy equivalent to one taken during a power cut, with the same consistency problems as in the next section.

This is the central warning of the lesson. A snapshot photographs the disk's blocks, not the application's logical state. And a running database has, at any instant, modified pages in memory that have not yet been written to disk, half-committed transactions, and WAL writes that the operating system has in its cache and has not flushed.

The resulting snapshot is equivalent to the state of the disk after a power cut. The good news is that PostgreSQL is prepared for that: on starting up on that volume it runs a WAL recovery and reaches a coherent state. The bad news is twofold: that recovery can take minutes on a large database, and you will lose the transactions that never reached disk. And with less robust engines, or with data spread across several volumes photographed separately, the result can be plainly useless. The professional solution is to coordinate the snapshot with the application: leave it in a consistent state just before, and release it just after.

Kubernetes offers the mechanism for the third: the pre and post hooks that run a command inside the container before and after the snapshot. It is not a native capability of the VolumeSnapshot, but of the backup tools, and that is why it is studied in the next lesson, 05-06, with Velero.

It is the most important statement of the module and the conceptual mistake that has destroyed the most data:

If the cloud account is deleted, if the zone is lost, if someone with administrator credentials destroys the project, the snapshots go with everything else. They are a magnificent tool for what they are — undoing a local change, quickly, in minutes — but they are not a disaster recovery strategy. Rutas Norte rule: snapshots are the short-term safety net — before a migration, a risky deployment, touching production — with a retention of days; the real backup goes to an external object store and is what gets built in the next lesson.

Tips:

1. Before any irreversible change — schema migration, major engine upgrade, clean-up script — take a snapshot and annotate it with the ticket. It costs seconds and it has saved careers.
2. Verify that a snapshot works, not that it exists. An untested snapshot is nothing. Restore periodically into a test PVC and check the rows.
3. Watch how full it is getting well in advance with a df -h over the mount point, and automate it with alerts at 75% in 07-04.
4. Diagnose storage by phase. If a pod does not start because of the volume, look in this order: kubectl describe pvc (provision), kubectl get volumeattachments (attach), kubectl describe pod (mount) and the logs of the CSI driver pod in kube-system.

With the hostpath CSI driver installed and rutasnorte-fast recreated with allowVolumeExpansion: true:

1. Check the current size of the bookings-postgres volume from inside the container.
2. Grow it from 10 GiB to 20 GiB by editing the PVC.
3. Follow the PVC's conditions until it finishes and verify the size with df -h.
4. Try to shrink it to 5 GiB and transcribe the error.
5. Check that the bookings are still there.

Rehearse the complete procedure you will use in production:

1. Insert five fictitious bookings into bookings-postgres.
2. Create a VolumeSnapshot called snap-practice with the reason, owner and ticket annotations. Wait for READYTOUSE: true.
3. Trigger the disaster: DROP TABLE bookings.
4. Create a restored PVC with dataSource and point the Deployment at it.
5. Verify that the five bookings are back.
6. Measure the total time from step 3 to step 5 and note it down: it is your real RTO for this kind of incident.

Answer with reasoning, in the form of a decision table you could present to the team:

• A. Which Rutas Norte volumes deserve automatic snapshots and which do not? Justify each component.
• B. With what frequency and what retention? Bear in mind that over a bank-holiday weekend about 400 bookings are sold per hour and that a 20 GiB snapshot costs money every month.
• C. Why are snapshots not enough as a data protection strategy for Rutas Norte? List three specific scenarios in which they would not save the situation.

Exercise 1

The final df -h should say 20G with only 1% used. If status.conditions shows FileSystemResizePending=True, the second stage is missing: a kubectl rollout restart deploy/bookings-postgres and then it will say so.

Expansion does not touch the data: it only grows the device and extends the filesystem over the new space.

Exercise 2

That number — of the order of one to three minutes on a local cluster, and five to fifteen in production with large volumes — is your RTO for this kind of incident. Note it down: it is the figure to bring to the recovery objectives conversation of 05-06. And notice that the original volume is still there, with the table dropped, available for investigating what happened.

Exercise 3

A. What deserves snapshots:

B. Frequency and retention:

The arithmetic that justifies point three: with a daily snapshot, the RPO is 24 h and over a bank-holiday weekend that is some 9,600 bookings lost. With a snapshot every 4 h, the RPO drops to 4 h, that is, about 1,600. And with continuously archived WAL it would drop to minutes, which is the real solution and is discussed in 05-06. The RPO is a business decision, not a technical one: somebody has to decide how many bookings it is acceptable to lose.

C. Three scenarios in which snapshots do not save the situation:

1. Loss of the storage system or of the zone. Snapshots live in the same array or region as the original volumes. If it disappears, they disappear with it. A backup in an object store in another region does survive.
2. Deletion of the cloud account or project, accidental or malicious. A compromised administrator, or a billing error that suspends the account, takes volumes and snapshots alike. The protection is a copy in another account, with different credentials and immutability enabled.
3. Logical corruption not detected in time. An application bug that corrupts records little by little over three weeks makes 7-day snapshots useless: they all already contain the corruption. You need copies with long retention — monthly, yearly — and, above all, verifiable logical dumps: a pg_dump that restores correctly proves the data is coherent, something a block snapshot does not prove. A fourth scenario worth mentioning: migrating to another provider or restoring on another cluster; an EBS snapshot does not restore on GKE, a logical dump does.

You have opened the black box. You know that CSI exists because having each vendor's code inside Kubernetes was unsustainable, and that its architecture is two halves: a controller plugin, deployed as a Deployment, that talks to the provider's API and creates, attaches and photographs volumes; and a node plugin, deployed as a DaemonSet, that formats and mounts on each machine. Between the driver and the Kubernetes API sit the sidecars — external-provisioner, external-attacher, external-resizer, external-snapshotter, node-driver-registrar —, which translate API objects into gRPC calls and let a vendor write only the logic of its storage. And you know the objects that describe that infrastructure: CSIDriver, with the declared capabilities, and CSINode, with each node's drivers and that Count which explains exceed max volume count. Above all, you have the complete flow in your head — Provision, Attach, NodeStage, NodePublish — and you know which symptom each phase produces when it fails, including the kubectl get volumeattachments that almost nobody uses and that tells a controller problem from a node one in a second.

You have expanded the bookings-postgres volume live from 10 to 20 GiB, with the database running and without touching a single piece of data. You know the three requirements — allowVolumeExpansion, driver support and a Bound PVC —, that the request is made always on the PVC and never on the PV, and the two stages of the process: the one on the real disk and the one on the filesystem, with the FileSystemResizePending condition that betrays when the pod has to be recreated for df -h to tell the truth. And you have the restriction etched in: it cannot be shrunk, so the size is thought through beforehand.

And you have mastered snapshots, with their object model copied from the volumes one — VolumeSnapshotClass, VolumeSnapshot, VolumeSnapshotContent, exactly the roles of StorageClass, PVC and PV —, their prerequisites that do not come with Kubernetes (the CRDs and the snapshot-controller), and the real workflow: a snapshot annotated with the reason, owner and ticket before a schema migration; the disaster; and the restore creating a new PVC with dataSource, which does not overwrite the original and preserves the evidence. You know how to clone a PVC live by changing the dataSource's kind, and that such a production clone carries real personal data and deserves the same care as the original.

And you take away the two warnings that give the next lesson its purpose. The first: a snapshot of a disk with a running database is crash-consistent, equivalent to the state after a power cut; PostgreSQL recovers via WAL, but it takes time and may lose what was not flushed, and the remedy is to coordinate the application with hooks before and after. The second, the most important in the module: a snapshot is not a backup. It lives in the same storage system as the original, so it does not survive the loss of the zone, nor the deletion of the account, nor logical corruption discovered three weeks late, nor is it any use for restoring on another cluster. It is a short-term safety net, excellent for what it is.

What is missing, therefore, is what really protects Rutas Norte: getting the data out of the cluster, putting it in an independent store, including the API objects too, coordinating the database so that the copy is consistent, defining retention and — this is non-negotiable — testing the restore. That is Backup and Restore of Persistent Data, the lesson that closes the module.