We close the storage module with a promise still outstanding. The Rutas Norte platform already has verified backups and bookings that survive the deletion of a pod, but bookings-postgres is still a single-replica Deployment with strategy: Recreate. That configuration is not an oversight: it is the only one that works, because a Deployment applies the same pod template — and therefore the same PersistentVolumeClaim — to all of its replicas. If we go up to two, both try to mount the same ReadWriteOnce volume and the second one waits forever.

The underlying problem is that a Deployment treats its pods as interchangeable cattle: any of them will do, the name does not matter, the order does not matter, it does not matter which disk it gets. For a database that is exactly the opposite of what we need. A PostgreSQL replica needs to know who it is, it needs its own disk and it needs to start in a particular order relative to the others.

The StatefulSet is the controller that provides those three guarantees. In this lesson we will study it in depth, migrate bookings-postgres from Deployment to StatefulSet without losing the existing volume, and finish with an honest warning that will shape the rest of the module: a StatefulSet gives you identity and a disk, but it does not replicate data for you.

1. What a StatefulSet guarantees that a Deployment cannot
2. Stable identity: ordinal names and per-pod DNS
3. serviceName and the headless Service
4. volumeClaimTemplates: one disk per replica
5. Start-up and shutdown order: podManagementPolicy
6. Update strategies: RollingUpdate, partition and OnDelete
7. Migrating bookings-postgres to a StatefulSet in practice
8. Scaling and what happens to the PVCs
9. The honest limit: identity is not replication
10. A second example: redis-cache

A StatefulSet is, like the Deployment, a controller that maintains a desired number of pods from a template. The difference lies in what it promises about those pods.

The three guarantees, stated precisely:

• Stable network identity. Each pod is given an ordinal name derived from the name of the StatefulSet, and a hostname that matches it. That name does not change even if the pod restarts, is recreated or ends up on another node.
• Stable per-replica storage. Each pod gets its own PVC generated from a template. The pod bookings-postgres-0 always mounts the volume belonging to bookings-postgres-0, never the one belonging to -1.
• Ordered deployment and scaling. Pods are created and removed in sequence, waiting for each one to be ready before continuing (if configured that way).

A warning up front that saves a lot of grief: using a StatefulSet does not make an application distributed. If your application does not know what to do with three copies of itself, three replicas in a StatefulSet only give you three isolated instances stepping on each other. We will come back to this in section 9.

A StatefulSet called bookings-postgres with replicas: 3 produces exactly these pods:

The suffix is an ordinal that starts at zero and is contiguous. There are no gaps: if -2 exists, then -0 and -1 exist. If you delete bookings-postgres-1 by hand, the controller recreates it with exactly the same name and mounts exactly the same PVC on it again. From the application's point of view, the pod "has restarted", not "has been replaced by another one".

That ordinal shows up in three places the application can consult:

• The pod name (metadata.name), reachable through the Downward API as we saw in module 3.
• The hostname inside the container, which matches the pod name.
• The subdomain that forms the DNS record, if there is a headless Service.

This is what lets a start-up script work out its own role:

That pattern — deriving the role from the ordinal — is the basis of many Kubernetes-ready database images. It is also, as we shall see, fragile: if pod -0 dies and another one has to take over, the ordinal no longer tells the truth.

The serviceName field is mandatory in a StatefulSet and points to the name of a headless Service, the kind we defined in lesson 04-02 with clusterIP: None and whose real use we said would arrive here.

Let us recall why. A normal Service has a virtual IP and spreads traffic across all the pods matching its selector. That is precisely what we do not want for a replicated database: if I write to "the service IP", I do not know whether I am writing to the primary or to a read-only replica.

A headless Service creates no virtual IP and no load-balancing rules. Instead, CoreDNS publishes one record per pod:

With this Service, each pod of the StatefulSet gets its own FQDN of the form:

That is:

Each one resolves to that pod's actual IP, not to a virtual IP. We can check it:

In addition, querying the name of the headless Service returns all the IPs of the ready pods, which is how you discover the membership list:

In practice you use two Services over the same StatefulSet:

The normal ClusterIP is the one the other components have known since module 4; there is no need to touch the configuration of bookings-api.

Here is the piece that breaks through the ceiling we hit in 05-03. Instead of referencing an existing PVC, the StatefulSet declares a PVC template, and the controller creates one per replica.

The name of the resulting PVC follows a fixed rule:

With the template called data and the StatefulSet bookings-postgres:

Inside the pod template, the volume is mounted by referring to the template name, not to the name of the actual PVC:

And now the crucial fact, the one that prevents the most grief and causes the most grief depending on how you look at it:

The PVCs created by volumeClaimTemplates are NOT deleted automatically. Neither when you scale down, nor when you delete the whole StatefulSet.

It is a deliberate design decision: data is the most valuable thing there is and Kubernetes does not throw it away for you. If you delete the bookings-postgres StatefulSet and create it again with the same name and the same template, the pods reattach to the volumes that already existed. That makes deleting and recreating a StatefulSet a safe operation, something we will take advantage of in the migration.

Since Kubernetes 1.27 there is a stable field for tuning that behaviour:

For bookings-postgres we will leave both on Retain, which is also consistent with the rutasnorte-fast class, whose reclaim policy is Retain. For a cache such as redis-cache, Delete is defensible.

By default (podManagementPolicy: OrderedReady), the StatefulSet creates pod -0, waits for it to be Running and Ready, and only then creates -1. When deleting or scaling down it goes the other way: highest ordinal first.

This is exactly what a database needs when its replicas connect to the primary: there is no point in starting the replica before the node it is going to copy from.

But it has a cost. If pod -0 never reaches Ready — badly written image, misconfigured probe, unavailable disk — the others are never created. A StatefulSet stuck on ordinal 0 is one of the most common incidents.

An important note: podManagementPolicy is immutable. It cannot be changed on an existing StatefulSet; you have to delete it (with --cascade=orphan if you do not want to touch the pods) and recreate it.

When you change the pod template — for example, moving from postgres:16.4 to postgres:16.6 — the StatefulSet applies its updateStrategy.

It updates pods in reverse ordinal order: -2 first, then -1, and -0 last. It waits for each one to be Ready before moving on to the next.

The reverse order is intentional: if -0 is the primary, it is updated last, minimising the time the cluster spends without a primary. And since each pod keeps its disk, the updated pod comes back with its data intact.

Inside RollingUpdate there is a field that turns the update into a phased rollout:

The rule is simple: only pods whose ordinal is greater than or equal to partition are updated. With partition: 2 and three replicas, only bookings-postgres-2 is updated. Pods -0 and -1 stay on the old version even though you changed the template.

This allows a manual, controlled canary release:

If pod -2 fails at step 3, it is enough to raise partition again and revert the image: the other two were never touched.

The controller updates nothing automatically. Pods are recreated with the new template only when you delete them by hand. It is the maximum possible control and it is used when the update requires external steps (migrating the schema, manually promoting a primary, verifying data integrity between one pod and the next).

This is the central exercise of the lesson: moving from a single-replica Deployment with a PVC called bookings-postgres-data to a StatefulSet, without losing the data.

The obstacle is naming. The StatefulSet will demand a PVC called data-bookings-postgres-0, whereas the existing one is called bookings-postgres-data. PVCs cannot be renamed. But we can reuse the underlying PersistentVolume, which is where the data really lives.

First of all, and with what we learned in 05-06:

The migration involves downtime. bookings-postgres is single-node: there is no way to do it live.

With the rutasnorte-fast class it is already Retain, but checking is free and getting it wrong is irreversible.

The PV is left in Released: it keeps the data but will not accept a new PVC until the reference to the previous one is cleared.

The PV is now back to Available.

volumeName forces the match with that specific PV, instead of provisioning a new, empty one.

Points worth highlighting in this manifest:

• serviceName points to the headless Service, not to the normal ClusterIP. Confusing the two is a frequent mistake.
• The resources keep the Guaranteed class we settled on in 03-05: requests equal to limits, 1 CPU and 2 GiB.
• The dedicated ServiceAccount and the automountServiceAccountToken: false from 03-06 are preserved.
• PGDATA points to a subdirectory of the mount point. PostgreSQL refuses to initialise over a directory containing lost+found, which is common on formatted block volumes.
• The selector uses only app and environment, following the project convention. And remember: spec.selector is immutable here too.

The data is still there. The old Deployment can now be deleted:

Scaling a StatefulSet is syntactically identical to scaling a Deployment:

When scaling up, the controller creates -1, waits for it to be Ready, then creates -2. Each one triggers the creation of a new PVC from the template, and with rutasnorte-fast and volumeBindingMode: WaitForFirstConsumer the volume is provisioned in the zone where the scheduler placed the pod.

When scaling down to one replica:

Pods -1 and -2 are gone, but their PVCs are still there (with whenScaled: Retain). Practical consequences:

• Advantage: if you scale back up to 3, the pods get their disks back with their data. Scaling is reversible.
• Cost: those volumes keep being billed. With a cloud provider, three 20 GiB SSDs cost the same whether they are mounted or not.

Cleaning up, when you are sure, is manual and explicit:

With the rutasnorte-fast class on Retain, the PV will move to Released and will keep the data until an administrator removes it. That friction is deliberate.

We reach the most important point of the lesson, and the one most often misunderstood.

We have scaled bookings-postgres to three replicas. That is not a PostgreSQL cluster. They are three PostgreSQL processes, each with its own empty, independent disk, with no relationship between them. If bookings-api wrote against the ClusterIP Service, bookings would land randomly in one of the three databases and the data would end up scattered and inconsistent.

What a StatefulSet gives you and what it does not:

Replication has to be implemented by the application. In PostgreSQL that means configuring streaming replication, creating the replication user, running pg_basebackup on the replicas, keeping a correct primary_conninfo and, above all, having something that decides on and carries out the promotion when the primary dies.

That "something" can be written by hand: init containers that detect the ordinal, probes that check the role, a script that rewrites the Services. It is exactly the work that operators do, and that is why lesson 06-07 will close the module by replacing this hand-rolled StatefulSet with a PostgreSQL operator that does know how to do all of that.

A practical rule for your day-to-day work:

If the application already knows how to run in a cluster (Kafka, Cassandra, etcd, Elasticsearch, ZooKeeper), a well-configured StatefulSet is enough. If it does not (PostgreSQL, MySQL, MongoDB in simple mode), a StatefulSet gives you the infrastructure but you need an operator or a managed service to supply the logic.

In the meantime, bookings-postgres in production stays on a single replica. That is honest: one well-backed-up, restorable replica is better than three replicas pretending to be a cluster.

redis-cache is a different and instructive case. It holds seat availability, a piece of data that is rebuildable from bookings-postgres, and losing it only causes a temporary spike in queries.

Does it deserve a StatefulSet? It depends what you want:

• Redis as a pure cache, with no persistence: a Deployment is perfectly valid. There is no state to preserve.
• Redis with AOF/RDB persistence so it does not start cold after a restart: a StatefulSet, because we want the pod to find its file again.

Rutas Norte chooses the second option for production, because after a restart on a bank-holiday weekend an empty cache pushes the load onto the database at exactly the worst moment.

Deliberate differences from bookings-postgres, and the reasoning behind each one:

Here we really do have three useful replicas, because each instance serves its share of the traffic independently and bookings-api can spread queries across them through the ClusterIP. There is no consistency to guarantee: if an instance does not have the key, the database is queried and the key is populated.

Confusing serviceName with the normal Service. serviceName must point to the headless Service. If it points to a normal ClusterIP, the StatefulSet starts without complaining but the per-pod DNS names do not work, and replicas trying to find each other fail in bewildering ways.

Forgetting to create the headless Service. Kubernetes does not validate that the Service named in serviceName exists. The pods start and everything looks fine until something tries to resolve bookings-postgres-1.bookings-postgres-nodes... and gets NXDOMAIN.

Expecting deleting the StatefulSet to delete the disks. With the default policy (Retain) it does not. After several rounds of testing you end up with dozens of orphan PVCs eating into your quota. Review them periodically: kubectl get pvc -A | grep -v Bound is not enough, because they stay Bound. Search by label and compare against the existing StatefulSets.

Scaling a database and believing it is now replicated. The most expensive conceptual mistake in this lesson. Three PostgreSQL pods with three empty disks are three different databases. Reread section 9.

A StatefulSet stuck on ordinal 0. With OrderedReady, if -0 never reaches Ready, nothing progresses. Diagnosis: kubectl describe pod bookings-postgres-0 and kubectl get events -n <ns> --sort-by=.lastTimestamp. Typical causes: a PVC in Pending because no node offers the required storage class, or a badly configured readiness probe.

Changing volumeClaimTemplates on an existing StatefulSet. It is a practically immutable field: only resources.requests.storage may be modified (and since 1.31, with the corresponding feature gate). Any other change causes a validation error. If you need to change the StorageClass, you have to migrate as in section 7.

Resizing the disk. Do not edit volumeClaimTemplates expecting it to propagate to the PVCs that already exist. You have to expand each PVC individually (kubectl patch pvc data-bookings-postgres-0 ...) as we saw in 05-05, and update the template so that future replicas are born with the new size.

Tip: always use subPath with PostgreSQL. Block volumes formatted as ext4 come with a lost+found directory at the root, and initdb refuses to work on a non-empty directory. Mounting at /var/lib/postgresql/data with PGDATA=/var/lib/postgresql/data/pgdata solves the problem cleanly.

Tip: kubectl delete statefulset X --cascade=orphan. It deletes the StatefulSet leaving the pods alive. This is the manoeuvre for changing immutable fields (selector, podManagementPolicy, serviceName) without interrupting the service: you delete the object, apply the corrected manifest and the controller adopts the existing pods, just as we saw with ReplicaSets in 02-02.

In the rutas-norte-dev namespace of your minikube (profile rutas-norte), deploy a StatefulSet called notes-demo with 3 replicas of busybox:1.36, a headless Service called notes-demo-nodes and a 100 MiB volumeClaimTemplates with the rutasnorte-standard class, mounted at /data. Each pod must write its name to /data/who-am-i.txt on start-up and then sleep.

Then:

1. Check the names of the pods and of the generated PVCs.
2. Write some extra content in pod notes-demo-1.
3. Delete that pod and verify that the new pod recovers the file.

Using the StatefulSet from exercise 1, change the image to busybox:1.37 with partition so that only the pod with ordinal 2 is updated. Verify that notes-demo-0 and notes-demo-1 still run the old image. Then complete the update.

With the StatefulSet from exercise 1 at 3 replicas:

1. Scale it down to 1 replica and check how many PVCs remain.
2. Scale back up to 3 and verify that notes-demo-2 recovers its file, not a new one.
3. Set persistentVolumeClaimRetentionPolicy.whenScaled: Delete, scale down to 1 again and check the difference.
4. Clean everything up.

Solution 1

The PVCs follow the pattern <template>-<statefulset>-<ordinal>, exactly as anticipated.

The file keeps the two previous lines and adds a third one from the new start-up. The pod has the same name and has found its disk again: stable identity and stable storage, exactly what a Deployment cannot promise.

Solution 2

Only the highest ordinal has been updated. If this were a database with the primary on -0, we would have tested the new version on the least critical replica.

The order observed in the events will be -1 first and -0 after: always descending.

Solution 3

One pod, three PVCs. It is the default behaviour (whenScaled: Retain) and the usual cause of unexplained storage bills.

Two lines: the one from the original start-up and the one from the new one. The disk is the same as before.

The PVCs of -1 and -2 have now been deleted along with their pods. For a cache that is what you want; for bookings-postgres it would be a quick way to lose a year of bookings.

The StatefulSet solves the ceiling we had been dragging since lesson 05-03. With it, each replica has a stable ordinal name, its own PVC generated from volumeClaimTemplates and an individual DNS name thanks to the headless Service that serviceName demands. Start-up and shutdown are ordered (OrderedReady) or simultaneous (Parallel) as suits you, and updates work down in reverse order with the option of holding them back in phases through partition.

We have migrated bookings-postgres from Deployment to StatefulSet by reusing the existing PersistentVolume — the trick is to clear the claimRef and create a PVC with the exact name the controller expects — and we have turned redis-cache into a three-replica StatefulSet with opposite, well-reasoned decisions in every field.

And we have been honest about the limit: the StatefulSet provides infrastructure, not logic. PostgreSQL does not replicate itself just by living inside one. That gap is what will motivate the operators of lesson 06-07, which will close this module by replacing our hand-rolled StatefulSet with software that knows how to promote a primary, create read replicas and recover to a specific point in time.

Before that there are other ways of running workloads we do not know yet. So far all of our workloads are spread wherever the scheduler wants and in whatever number of copies we ask for. But there is a category of process — infrastructure agents: log collectors, metrics exporters, network plugins — for which "how many replicas" is the wrong question: the right answer is always one per node, no more and no fewer, and automatically on the nodes added tomorrow. That is the subject of the next lesson: DaemonSets.