In the previous lesson we took web-store and bookings-api to production. It was thirteen objects and a long checklist, but the margin for error was generous: if a pod is lost, an identical one is born and nobody notices. Here that safety net disappears. bookings-postgres holds the personal data of Rutas Norte's customers, their bookings and their payments. A lost pod can mean minutes of downtime; a lost volume, an existential problem for the company.

This lesson works through the complete scenario of a stateful workload in production: from the uncomfortable question of whether that database should be in Kubernetes at all, to the timed restore procedure, by way of failover, major-version upgrades and the very different case of redis-cache, where losing the data is acceptable.

Compliance warning. Everything described here affects customers' personal data (name, email, travel history, tokenised payment details) and business continuity. Decisions about where backups are located, retention periods, encryption, staff access to the data and cross-region transfers must be reviewed and approved by the organisation's compliance officer before being applied. The values in this material are illustrative and do not replace that review.

1. The prior question: should the database live in Kubernetes?
2. What makes a stateful workload special
3. bookings-postgres with CloudNativePG
4. How bookings-api connects: the connection pooler
5. Failover: simulating it and measuring it
6. Backups and point-in-time recovery
7. Major-version PostgreSQL upgrades
8. Read replicas for occupancy-reports
9. redis-cache: when losing the data is acceptable
10. Checklist for a stateful workload in production

It is tempting to answer "yes" for the sake of consistency: if everything else is in the cluster, so should the database be. That is a bad reason. The right question is which option minimises total risk at a cost the company can bear.

There are three real options.

The decision criterion, in one sentence: use the managed database unless you have a concrete reason not to, and never build a hand-rolled StatefulSet for a production database.

Concrete reasons that justify the operator inside the cluster:

• The managed bill is out of proportion to the size of the company.
• You need extensions or parameters the provider does not allow.
• There is a requirement for portability between clouds or deployment in your own data centre.
• The team already has demonstrable operational maturity on Kubernetes and someone with real PostgreSQL knowledge.

Reasons that justify nothing: "it looks tidier", "this way everything is Kubernetes", "the YAML is pretty".

Rutas Norte's decision. In module 6 CloudNativePG was introduced as a replacement for the hand-rolled StatefulSet, and in module 10 EKS in eu-west-1 was chosen. The platform team has decided to keep bookings-postgres inside the cluster with CloudNativePG, with two explicit conditions written into the minutes: that backups always go to an object store outside the cluster, and that the restore is tested quarterly. The reason for the decision is not technical but economic and about portability: the data volume is modest (around 120 GB), the equivalent managed instance with multi-zone high availability would cost roughly three times as much, and the board wants to keep the option of moving the platform to another cloud in two years' time.

It is a legitimate decision with one clear consequence: the platform team takes on the database on-call rota. If nobody is willing to sign up to that, the right answer was the managed one.

Four properties that break the comfortable assumptions of the previous module.

A bookings-api pod is interchangeable with any other. A PostgreSQL pod is not: one is the primary and accepts writes, the rest are replicas and only read. The identity is not in the label, it is in the state of replication at that instant, and it can change without anyone deploying anything.

When starting a PostgreSQL cluster, the first instance must initialise the data directory; the others must clone from it. When upgrading, you have to upgrade the replicas first and promote afterwards. RollingUpdate knows nothing about that.

The volume is not a cache: it is the asset. This changes three things at once:

• The PV's reclaim policy must be Retain, not Delete (05-02).
• Deleting the resource must not delete the data: explicit protection is needed.
• Backups are neither optional nor an operational detail: they are part of the design.

The practical conclusion: for anything stateful, you do not deploy workloads, you delegate to an operator that understands that specific database. That is exactly what we saw in 06-07 and what we are going to apply now.

CloudNativePG is an operator that implements a PostgreSQL cluster with streaming replication, primary election, continuous backups and controlled upgrades. It does not use StatefulSets: it manages the pods directly because it needs finer control over ordering.

The decisions worth understanding:

• instances: 3 and not 2. With two instances, as soon as one goes down you are left with no redundancy exactly when you need it most, because rebuilding a 120 GB replica takes a while. Three is the minimum you can defend in production.
• primaryUpdateMethod: switchover. When a minor upgrade is applied, the operator upgrades the replicas first, then promotes an already-upgraded replica and finally upgrades the old primary. The outage is measured in seconds, not minutes.
• Separate walStorage. The WAL has a very different write pattern from the data and, above all, if the data volume fills up because of the WAL, PostgreSQL stops. Separating them turns a high-severity incident into a minor one.
• podAntiAffinityType: required. With preferred, a tightly packed cluster can place two instances on the same node and high availability becomes fictional.
• inheritFromIAMRole: true. No access key stored anywhere: the pod's ServiceAccount has a cloud role attached. It is the direct continuation of what we saw in 08-01 and 10-06.
• retentionPolicy: "30d". This value must be validated by the compliance officer: retaining too little breaches accounting and continuity obligations; retaining too much, with personal data inside, breaches the storage-limitation principle.

The Cluster's backup section defines where and how; you also need to say when the base backup is taken.

With the daily base backup and continuous WAL archiving, the data-loss window is seconds rather than a day: you restore the most recent base backup and replay the WAL up to the desired instant. That is the point-in-time recovery of section 6.

The operator keeps one pod as the primary and records its identity in the Cluster status. When the primary stops responding to the checks for longer than failoverDelay, the operator picks the replica with the most advanced WAL, promotes it and reconfigures the others to follow the new one. All of that without a single manifest changing.

The Services it creates automatically:

PostgreSQL creates one process per connection. With max_connections: 200 and an HPA that over the May bank-holiday weekend takes bookings-api to 40 replicas with a pool of 20 connections each, the arithmetic is devastating: 800 connections requested against 200 available. The result is not slowness, it is connections being refused and 500 errors on ticket sales.

The solution is a connection pooler (PgBouncer) in front of the database, which CloudNativePG manages as a resource of its own:

A thousand application connections are multiplexed over forty real ones. That is why the bookings-api ConfigMap from the previous lesson points to bookings-postgres-pooler-rw and not directly to the -rw Service.

The price to pay: poolMode: transaction is incompatible with session-bound features (named prepared statements in some drivers, LISTEN/NOTIFY, session temporary tables, persistent SET). In bookings-api it was verified that none of them is used. If any were, the alternative is session, which greatly reduces the pooler's advantage.

A high-availability mechanism that has never been tested is a hypothesis. This is the test Rutas Norte runs in rutas-norte-pre before every peak season.

Results from the latest drill in rutas-norte-pre (three repetitions, mean values):

Fifteen seconds unable to write. If bookings-api does nothing about it, those fifteen seconds are fifteen seconds of 500 errors on ticket purchases, and in the middle of the May bank-holiday weekend that is several hundred lost sales.

Three mechanisms, and the order matters.

Bounded timeouts. Without a timeout, a connection to a dead primary hangs until the operating system's timeout (minutes). With one, it fails fast and can be retried.

Retries with exponential backoff and jitter, only for what is idempotent. Retrying a SELECT is safe. Retrying INSERT INTO bookings without more thought can duplicate the booking and charge the customer twice: an idempotency key is required.

With four attempts and a 200 ms base, the window covered is around 4-6 seconds. It does not cover the full fifteen seconds of the outage, and that is deliberate: stretching it further would make requests pile up on the server until memory ran out.

A circuit breaker for what cannot be retried. When failures exceed a threshold, the circuit opens and the API stops trying for a few seconds, returning an immediate degraded response instead of accumulating hung requests.

The combination of the three reduces the real impact of a failover to a few seconds of partial degradation, with sales recovering on their own. It is the difference between a severity-1 incident and a note in the log.

With continuous WAL archiving, the real RPO is seconds. The RTO is the one you have to measure, and it is only measured by restoring.

CloudNativePG restores by creating a new Cluster from the object store. You never restore "on top of" the existing cluster: you bring up a parallel one, verify it and then decide.

Verification before accepting the restore as good:

Drill of 12 July 2026 on a 118 GB backup:

Forty-three minutes, within the one-hour RTO, but with little margin. Two actions came out of the drill: raising maxParallel from 4 to 8 for recovery (already applied above) and keeping the restore manifest written and versioned in the repository, with the target instant as the only parameter to fill in. The six minutes of "writing the manifest" under pressure are the worst possible place to improvise.

A backup that has never been restored does not exist. That is a literal statement, not a rhetorical figure. The real failure modes are mundane: the store credential expired four months ago and nobody looked at the alert; the bucket had a lifecycle policy that moved objects to cold storage with hours of retrieval latency; the backup was being taken, but of a database that was no longer the production one. All of them are caught by restoring, and none of them is caught by checking that the ScheduledBackup is green.

Rutas Norte's rule: a full timed restore every quarter, with minutes taken and the time written down. It is in the maintenance calendar in 11-06.

Minor upgrades (16.4 → 16.6) are handled by the operator on its own: you change imageName, it upgrades the replicas, switches over and upgrades the old primary. An outage of seconds.

Major ones (16 → 17) are another matter: the data directory format changes, streaming replication does not work between different versions and rolling back is not trivial. There are three routes.

Rutas Norte chose logical replication. The procedure, in summary:

Sequence on the day of the change:

1. Weeks before: bring up the 17 cluster in rutas-norte-pre, run the full bookings-api test suite against it and compare execution plans for the ten most expensive queries. A change of plan in the planner is the real risk of a major upgrade.
2. Days before: bring up 17 in pro and leave it syncing by logical replication until the lag is down to milliseconds.
3. Cutover window (about 90 s): put bookings-api into read-only mode via a feature flag, confirm zero lag, stop the subscription, point the ConfigMap at the 17 cluster's pooler, restart the deployment and remove read-only mode.
4. Afterwards: watch for 48 hours. The 16 cluster is kept for a week shut down but intact, as the rollback plan.

One detail that is always forgotten: a full ANALYZE after the migration. The planner statistics are not carried over, and without them queries run slowly for hours and it looks as though the upgrade went wrong.

occupancy-reports is the nightly CronJob that goes over months of bookings to calculate occupancy by route and by time slot. These are heavy queries which, run against the primary, compete with ticket sales.

The solution is straightforward: point it at the -ro Service, which only routes to replicas.

Two warnings about read replicas:

• Eventual consistency. A replica runs a few milliseconds behind. For reports that is irrelevant; for "I have just booked and I cannot see my booking" it is an error visible to the user. Rule at Rutas Norte: what the user has just written is read from the primary; everything else can go to a replica.
• Recovery conflicts. A long query on the replica can conflict with incoming WAL and be cancelled. It is tuned with max_standby_streaming_delay, accepting more replication lag during the reports in exchange.

Redis is a stateful workload too, but of a different category: its data is rebuildable. That difference changes everything.

At Rutas Norte, redis-cache holds three things:

The basket is the awkward case: it is not critical like a confirmed booking, but losing it is visible and annoying.

Rutas Norte's decision: AOF persistence enabled with appendfsync everysec, a single node with a persistent volume and no replica. The reasoning:

• Without persistence, every pod restart (a node upgrade, a configuration change) empties every active basket. That happens several times a month.
• With AOF every second, a clean restart loses at most one second of writes and the baskets survive.
• Setting up Redis Sentinel or a cluster for data with a fifteen-minute TTL is complexity with no return.

And, above all, the application assumes Redis may not be there: if the cache does not respond, bookings-api goes to PostgreSQL; if the basket is gone, the user is asked to redo their selection with a clear message. No Redis call may bring down a request.

One extra caution: if Redis goes down over the May bank-holiday weekend, the entire read load shifts to PostgreSQL all at once. You have to size the database to survive that scenario, or the failure of a dispensable component turns into the failure of one that is not.

Row 18 is not bureaucracy: in a PostgreSQL cluster holding personal data, backup retention, the region of the object store, who can open a psql session against production and what is recorded about it are decisions with legal consequences, not merely technical ones.

• Putting the database on Kubernetes for the sake of aesthetic consistency. If nobody on the team knows how to do a point-in-time recovery or is willing to be on call, the right answer is the managed database, even if it costs more.
• Trusting a green ScheduledBackup. The indicator says the process ran, not that the backup is restorable. Only a timed restore proves that.
• Backups in the same cluster or the same account that produces the data. An accidental deletion with broad permissions, or a credential compromise, takes the data and the backup at once. A separate store and, if possible, a separate account with immutability.
• Connecting the application to the -rw Service with no pooler. It works perfectly until the HPA scales, and then it fails on exactly the busiest sales day.
• Retrying non-idempotent writes. It duplicates bookings and charges. Every retryable write needs an idempotency key.
• Using allkeys-lru in a Redis that holds baskets. When memory fills up it evicts active baskets. volatile-lru, with a TTL only on what is cache, protects the data that does matter.
• Tip: write the restore manifest before you need it and keep it in the repository with the target instant as a blank to fill in. It saves the most expensive minutes of the incident.
• Tip: run ANALYZE after any migration or major upgrade. Without statistics the planner makes bad decisions and everything looks broken.
• Tip: size the database for the scenario in which the cache is absent. Otherwise a dispensable failure turns into a critical one.

Rutas Norte is about to launch a new product, freight-routes, with its own PostgreSQL database of around 8 GB, three developers, no dedicated platform team and a launch planned in six weeks. The data includes contact details for corporate customers. Recommend one of the three options and justify the decision with at least four criteria from the table in section 1. Also state what additional review is required because of the contact details.

bookings-api has an HPA with maxReplicas: 40 and each replica opens up to 20 connections. occupancy-reports opens 5 connections against the replicas. bookings-postgres has max_connections: 200, of which PostgreSQL reserves 3 for the superuser. Work out the maximum connections requested without a pooler, explain what happens at the May bank-holiday peak and check whether max_client_conn: 1000 and default_pool_size: 40 are appropriate values.

During the quarterly drill, the restore Cluster ends up like this:

List three possible causes, say how to tell them apart and what fix you would apply in each case. Also state what this implies about the real RPO.

Solution 1. Recommendation: the provider-managed database. Criteria: (a) staffing cost, there is no platform team and running the operator would fall on three developers who need to be building the product; (b) time to production, six weeks leaves no room to acquire operational maturity with CloudNativePG; (c) risk of catastrophic failure, with no on-call rota in place a night-time failure would go unanswered; (d) infrastructure cost, the provider's premium on 8 GB is small in absolute terms, quite unlike the case of bookings-postgres with 120 GB. The hand-rolled StatefulSet is ruled out from the start. Additional review: the contact details of corporate customers are personal data of the individuals acting as contacts, so the choice of region, the backup retention and the terms with the data processor (the cloud provider) must be reviewed by the compliance officer before signing the contract.

Solution 2. Without a pooler: 40 × 20 = 800 connections from bookings-api, plus 5 from reports = 805 requested against 197 usable. At the May bank-holiday peak, as soon as 197 is exceeded PostgreSQL refuses with FATAL: sorry, too many clients already; the affected pods fail their readiness probes, leave the load balancer, the HPA sees more load on the remaining ones and scales further, making the problem worse: a destructive feedback loop. With the pooler: max_client_conn: 1000 covers the 805 requested with margin (appropriate); default_pool_size: 40 plus reserve_pool_size: 10 opens at most 50 real connections against the primary, comfortably within 197, leaving room for reports, maintenance and the superuser. Both values are correct. It is worth watching the PgBouncer queue-wait metric: if it grows during the peak, the bottleneck becomes default_pool_size and it would need raising (with headroom up to around 150 real connections).

Solution 3. Possible causes: (a) insufficient retention, the WAL for that period has already been deleted by the 30-day policy or by a bucket lifecycle rule; you tell it apart by listing the wals/ prefix in the store and checking whether the segment exists; fix: choose a target instant within the available range and review the retention policy and the lifecycle rules. (b) Continuous archiving failure, the primary stopped archiving WAL at some point (expired credential, permissions, full disk) and there is a gap; you tell it apart by looking at cnpg_collector_last_failed_archive_time and the primary's logs; fix: repair the archiving and, very importantly, take a new base backup immediately. (c) Wrong permissions or path in the restore, the serverName or the destinationPath do not match those of the source, or the federated role has no read access to that prefix; you tell it apart because every segment would fail, not one specific one; fix: correct serverName/destinationPath and the role's permissions. Implication for the RPO: if the archiving has gaps, the declared RPO of 5 minutes is false, and the real one is the age of the last complete base backup, which can be nearly 24 hours. That is exactly why the PostgresBackupStale alert exists and why the quarterly drill is mandatory.

We have worked through, end to end, the scenario of a stateful workload in production. We started with the question best not dodged — whether the database should be in Kubernetes — and arrived at a reasoned decision with conditions attached: at Rutas Norte it stays inside the cluster with CloudNativePG, with backups outside and a restore tested every quarter. We saw what makes state special (identity, order, irreplaceable data, upgrades that do not tolerate RollingUpdate), the complete Cluster resource with its decisions justified, the connection pooler that stops the HPA's success from bringing down the database, the failover measured at fifteen seconds and the three mechanisms that let bookings-api ride through them with no visible errors, point-in-time recovery with its real time of 43 minutes, the major-version upgrade by logical replication, the read replicas for the reports and redis-cache as the case where losing the data is acceptable, provided the application accepts that.

The idea that sums the lesson up: with stateful workloads, what saves you is not the manifests but the tested procedures. The backup that was never restored, the failover that was never simulated and the upgrade that was never rehearsed in pre are debt waiting to fall due at the worst possible moment.

We now have the stateless application and the database in production. What we have not covered yet is how the code gets there. In the next lesson, CI/CD with Kubernetes, we follow the complete path from a developer's git push to the pod serving requests in rutas-norte-pro, by way of building the image, scanning and signing it, testing against an ephemeral cluster and promoting between environments with Argo CD.