We close the module with the lesson that turns Rutas Norte into a platform you can trust. We already have the data outside the container, on a volume that survives the pod, expandable and with snapshots. But the previous lesson ended with a warning that puts it all in perspective: a snapshot lives in the same storage system as the original, so it does not survive the loss of a zone, the deletion of an account or corruption discovered three weeks late. A real backup is something else: it leaves the cluster, goes to an independent system, includes the API objects too, is coordinated with the application to be consistent, has a defined retention and — this is the one non-negotiable — is tested by restoring it. In this lesson you will build the two pieces Rutas Norte needs: the scheduled logical dump of bookings-postgres and the backup of the complete platform with Velero, and you will run the disaster drill by deleting a whole environment and recovering it against the clock.

1. What really has to be saved in a cluster
2. A snapshot is not a backup: the 3-2-1 rule
3. RPO and RTO applied to Rutas Norte with numbers
4. Logical backup: pg_dump from a scheduled Job
5. Compression, encryption and restoring the dump
6. Velero: what it does and how it is built
7. Installation and the first backup of a namespace
8. pre and post hooks: the consistent backup
9. Scheduled backups with retention
10. Restoring into another namespace
11. The disaster drill
12. What does not restore on its own, and the minimal runbook
13. Retention, cost and personal data

The first question is not "how do I take backups" but "of what". In a Kubernetes cluster there are three different things, and each one needs a different technique:

Three observations that shape everything else. Git is not enough: the manifests let you recreate the shape of the platform, but they do not contain the Secrets (which we deliberately kept out of Git in 03-02), nor the certificates issued by cert-manager (04-05), and of course not the data. The etcd backup is the cluster administrator's business, not the application team's, and on a managed cluster you do not even have access to etcd: that is why the practical approach for Rutas Norte is to back up API objects per namespace, which is portable between clusters as a bonus. And what is truly irreplaceable is the data: a Deployment is recreated in seconds; a booking sold and paid for is not.

We return to the warning of 05-05 and give it operational shape with the classic rule, which is still the best summary. 3-2-1: at least 3 copies of the data, on 2 different media or systems, with 1 of them off the main site. Applied to Rutas Norte:

The fourth row is what turns the set into a strategy: without it, everything else disappears along with the infrastructure it protects. Two reinforcements considered mandatory today: immutability — the object store in lock mode (Object Lock / WORM), so that not even stolen credentials can delete or encrypt the backups — and separation of credentials: the account that writes the backups must not be able to delete them, and the destination must be in another account or subscription.

Two acronyms that must be kept well apart because they are constantly confused:

And now the numbers, which is what turns the conversation into something useful. Rutas Norte sells about 60 bookings per hour on a normal working day, and about 400 on a bank-holiday weekend or at the start of the holidays, with peaks of 700.

Every row implies a business conversation, not a technical one: how much does it cost to lose 9,600 bookings? It is not just the money; it is rebuilding by hand, handling complaints, customers turning up at the station with no ticket and reputational damage in the middle of peak season.

Rutas Norte's decision:

The RTO measured in the exercise of 05-05 — of the order of two minutes locally — was for restoring a volume. The real RTO of a disaster includes detecting, deciding, restoring, verifying and reopening the service. That is why it has to be measured with a drill (section 11), not estimated.

A logical backup is a file with the instructions needed to rebuild the database. Its advantages over a block snapshot are decisive:

They do not compete: they complement each other. The snapshot is fast for undoing; the dump is the copy that really proves the data is sound.

First, the volume where the dumps are written: k8s/base/postgres-backups-pvc.yaml, a 50 GiB PVC called postgres-backups, ReadWriteOnce, with the usual labels plus app.kubernetes.io/component: backups. It goes on the rutasnorte-fast class (with Retain) because if the backups are lost, the safety net disappears.

And now the CronJob. Note: the CronJob object is studied in depth in 06-03; here we use it as a tool, keeping only what is essential to read it.

The decisions inside it, one by one:

• concurrencyPolicy: Forbid: if a dump takes longer than an hour, the next one does not start. Two simultaneous pg_dump runs would double the load on the database exactly when it is already slow.
• set -euo pipefail and the verification with pg_restore --list: without the former, a failed pg_dump can leave a truncated file and the Job finish as Completed, giving you backups that are worthless; the latter checks that the dump is at least readable. Real verification is restoring it (section 11).
• --no-owner --no-privileges: the dump does not drag owners or permissions along, which allows restoring it on another cluster with another user. And with its own ServiceAccount (03-06) the pod has only what it needs, while the local 72 h retention turns the PVC into the hot copy for restoring fast; the long retention lives outside the cluster.

And you have to remember the NetworkPolicies of 04-06: the CronJob's pod is new in the namespace and deny-all will block it. Its conversation has to be authorised explicitly:

Verification:

The -Fc format compresses but does not encrypt. And that file contains the name, ID number, phone number and email address of every Rutas Norte customer: it is exactly the sort of artefact that cannot be left in the clear on any medium. The practical solution is to encrypt with a public key whose private key is not in the cluster:

The BACKUP_PUBLIC_KEY variable arrives by secretKeyRef from a backup-public-key Secret. And there is the important point: only the public key lives in the cluster, so whoever compromises the cluster can write backups but cannot read them. The private key is kept outside (a corporate secrets manager, a safe), and the procedure for accessing it must be documented, because a backup nobody can decrypt on the day of the disaster is not a backup.

You launch a helper pod with the postgres:16.4 image mounting the postgres-backups PVC at /backups (with kubectl run ... --overrides or a one-off Job), and from inside:

Always restore into a new database and verify before replacing. Restoring straight over the production database with --clean is the fastest way of turning a recoverable incident into an irreversible one.

pg_restore options that save time on the bad day:

--jobs=4 restores in parallel and greatly reduces the RTO on large databases; --table=bookings restores a single table, which is the most common case (someone dropped a table); --schema-only and --data-only separate structure from data; and --list with --use-list let you inspect the contents and restore only a part.

The logical dump protects the database's data. It does not protect the Secrets, nor the PVCs, nor the Ingresses, nor the certificates, nor the rest of the namespace's objects. That is what Velero is for, the standard tool for backing up and migrating Kubernetes resources.

Velero does three things. It backs up the API objects of a namespace (or of the whole cluster), filtered by labels or by type, to an object store (S3, Azure Blob, GCS, MinIO). It backs up the volume data in two ways: by asking the CSI driver for snapshots, or by copying file by file with Kopia/Restic to the same object store, which is what really gets the data out of the storage system. And it restores all of that, on the same cluster or another one, with namespace remapping and filters.

The distinction to be clear about from the start:

For Rutas Norte: CSI snapshots for the daily one (fast, for undoing) and a copy to the object store for the weekly long-retention backup, which is the one that satisfies the "1" of the 3-2-1 rule.

To practise on minikube we will use MinIO as an S3-compatible object store, deployed in the cluster itself. In production it would be a real bucket in another region and another account.

The first backup, of the pre-production namespace:

Note what it has captured: the Secrets, which are not in Git; the PVCs; the NetworkPolicies; the ServiceAccounts. And the volume data, 18.4 GB copied file by file to the object store.

The two volume variants are --snapshot-volumes=false (API objects only: blazingly fast, but with no data) and --snapshot-volumes=true (CSI snapshots: fast, but the data stays in the same storage system).

Here we solve the consistency problem we left open in 05-05. Velero's hooks run a command inside the container before and after backing up the volume, which allows the database to be left in a coherent state.

They are declared as annotations on the pod, or on the backup itself. The declarative way, in the Deployment's podTemplate:

What each one does. pg_backup_start (the name from PostgreSQL 15 onwards; it used to be pg_start_backup) puts the server into backup mode: it forces a checkpoint and guarantees that the files in the data directory form a restorable set, even while writes keep coming in. And pg_backup_stop closes that mode: it is essential that it always runs, because a database left in backup mode accumulates WAL without releasing it and ends up filling the disk; that is why the post hook runs even if the backup fails.

A simpler and often preferable alternative for medium-sized databases: the pre hook does a pg_dump straight into an emptyDir that Velero also backs up. It sacrifices some time in exchange for a verifiable logical dump.

A manual backup is good for learning; what protects you is a schedule with automatic retention, which Velero solves with Schedule and the TTL, which deletes the backup and its data on expiry.

The resulting Rutas Norte calendar:

Only the pro-weekly row satisfies the "1" of the 3-2-1 rule. The others are convenience and speed.

Mandatory monitoring: a backup that fails silently is worse than having no backups, because it creates unfounded confidence. It is spotted with velero backup get | grep -v Completed and investigated with velero backup describe <name> --details and velero backup logs <name> | grep -i error. That check must be an automatic alert, not a manual review; it is set up with the tools of 07-04.

Restoring over the original namespace in production is the most delicate operation. The correct practice is to restore somewhere else first and verify:

Restore options used every day:

And the warning that saves you a fright: by default Velero does NOT overwrite objects that already exist. If you restore over a live namespace, you will see a pile of "already exists" warnings and think the restore failed, when what it has done is protect you. For a real recovery, the target namespace must be empty or you have to use --existing-resource-policy=update deliberately.

An untested backup is not a backup. This is the compulsory exercise of the module, and at Rutas Norte it is run every quarter on rutas-norte-pre, against the clock and with a written record.

The drill record must capture, as a minimum: date and owner, backup used, measured RTO, effective RPO (the time elapsed between the backup and the disaster), what did not come back on its own, issues and follow-up actions. In the second exercise you will write it in full.

The "what did not come back on its own" section is the most valuable in the record. It is literally impossible to know without running the drill, and it is what turns a theoretical recovery into a real one.

A short document, in the repository, that someone on call at three in the morning can follow without thinking:

Backups cost money, and without automatic retention the spend grows unchecked. With bookings-postgres at around 20 GiB, the 72 hourly dumps take up about 30 GiB (they compress to ~410 MB each), the 7 daily snapshots about 25 GiB as they are incremental, and the 13 weekly backups about 260 GiB in the object store. Levers for tuning it: use cold storage classes for the old ones, take advantage of Kopia's deduplication, and do not back up what can be regenerated (redis-cache is in no backup at all, following what was decided in 05-03).

And here is what cannot be left as a technical detail. Rutas Norte's backups contain personal customer data: the name, ID number, phone number and email address of every person who has bought a ticket. That means each backup is a processing of personal data with the same duties as the original system, and in some respects more demanding ones, because backups multiply, replicate and get forgotten.

The minimum requirements Rutas Norte applies:

REGULATORY COMPLIANCE WARNING

Everything above is technical architecture, not legal advice. The retention periods, the geographical location of the backups, the handling of the right to erasure over data already backed up, the legal basis for the processing and the obligations in the event of a security breach must be reviewed and approved by the organisation's compliance and data protection officer before this configuration is put into production. The values in this course (72 hours, 7 days, 90 days) are teaching examples chosen to illustrate the mechanics of retention, not legal recommendations: the correct period depends on the applicable regulations, the sector and the declared purpose of each processing activity, and keeping personal data for longer than necessary is as much non-compliance as failing to protect it.

Tips:

1. Automate the verification, not just the backup. A weekly Job that restores the latest dump into a test namespace and compares the row count turns "we think we have backups" into "we know we have backups".
2. Document the real RPO and RTO, not the desired ones, and review them after every drill. They are the figures the business needs in order to decide how much to invest.
3. The backup must be restorable without the person who set it up. If the procedure lives in someone's head, it does not exist: write it in the runbook, in the repository. And when you hesitate between spending on backups or on anything else, spend on backups: it is the only component of the platform whose absence goes unnoticed until it is too late.

In rutas-norte-dev, set up the complete logical backup chain:

1. Create the 5 GiB postgres-backups PVC on the rutasnorte-fast class.
2. Adapt the CronJob of section 4 to dev, with schedule: "*/10 * * * *" so as not to wait an hour.
3. Trigger a manual run and check in the logs that the dump has been created and verified.
4. Drop the bookings table and restore it into a new database from the dump, without touching the original.
5. Check that the data matches and explain why restoring into a new database matters.

Run the complete drill of section 11 on rutas-norte-pre (create it with the five components if you do not have it) and write the record with: prior inventory, backup used, measured RTO, effective RPO, what did not come back on its own, issues and follow-up actions.

Then compare the measured RTO with the 8-hour target set for pre in section 3 and reason about whether the policy is adequate or has to change.

The Rutas Norte board asks: "if we lose the whole data centre tomorrow, how long until we can sell again and how many bookings do we lose?". Prepare the answer as a short document:

• A. A table of what is backed up, with what method, frequency, retention and what each backup protects against.
• B. The committed RPO and RTO, with the number of bookings lost in the worst case (bank-holiday weekend, 400 bookings/hour).
• C. The three elements that do not restore on their own and who is responsible for each.
• D. A section on personal data covering what the backups contain, how they are protected and what the compliance officer must validate.

Exercise 1

Why into a new database. Three reasons. First, if the dump were corrupt or incomplete, restoring over the top with --clean would have destroyed what was left: you would end up with neither the damaged data nor the backup. Second, the damaged database is the evidence for the post-mortem: without it you will not know what happened or since when. And third, it lets you compare before switching over: row count, maximum date, referential integrity. The final swap is a two-second rename, so the time saved by restoring over the top is negligible against the risk.

Exercise 2

A sample record:

Assessment of the policy: the target RTO for pre was 8 hours and the measured one is 23 minutes, so it is met with an enormous margin. But the useful conclusion is another one: the same procedure applied to production would not give 23 minutes, because in pro the volume is ten times bigger, DNS has to be updated with its propagation, external secrets regenerated and the business verified end to end. The correct action is to repeat the drill on a production backup restored into a separate namespace, which is what gives the real figure behind the committed 2 hours.

Exercise 3

A. What is backed up:

B. Commitments:

The last row is the answer to the board's question, and it is deliberately uncomfortable: the backup that survives losing the data centre is the weekly one, so in the worst case up to seven days of bookings would be lost. If that is not acceptable — and it should not be —, there are two investments that fix it: moving the external backup to daily (a 24 h RPO) and, above all, archiving PostgreSQL's WAL continuously to an external object store, which would bring the RPO down to minutes. It is the plan's top-priority improvement.

C. What does not restore on its own:

D. Personal data:

The backups contain the name, ID number, phone number and email address of every customer who has bought a ticket, plus the complete booking history. Protections applied: encryption at rest of the dumps with a public key (the private one kept outside the cluster), bucket encryption, encryption in transit, write-only credentials for the backup process, restore access limited by RBAC and logged, automatic retention by TTL and compulsory anonymisation of any clone destined for non-production environments.

What the compliance officer must validate, before any of this goes into production:

1. That the retention periods (72 h, 7 days, 90 days) are the legally correct ones for this purpose, no more and no less.
2. That the region of the backup store is within the permitted and declared geographical scope.
3. The procedure for the right to erasure: what is done with backups that already contain the data of whoever exercises it, and how it is documented.
4. The legal basis for the processing and its reflection in the record of processing activities, including the backups as processing.
5. The breach notification protocol if a backup is compromised.
6. The data processor terms with the cloud provider hosting the backups.

This document is a technical proposal and it does not replace that review.

Rutas Norte is now a platform you can trust. You know what has to be saved — the manifests, which are already in Git; the API state, with the Secrets and PVCs that Git does not contain; and the volume data, the only truly irreplaceable thing — and that each one demands a different technique. You have internalised the 3-2-1 rule and you know that, of all Rutas Norte's backups, only the weekly one to the object store in another region satisfies the "1"; the others are convenience and speed. And you handle RPO and RTO with concrete numbers: at 400 bookings per hour on a bank-holiday weekend, a daily backup means losing up to 9,600 bookings, and that turns a technical discussion into a business decision somebody has to take and sign off.

You have built the two pieces. The logical backup: an hourly CronJob that runs pg_dump -Fc, verifies the dump with pg_restore --list, encrypts it with a public key whose private counterpart does not live in the cluster, applies local retention and — the detail that is always forgotten — has its own NetworkPolicy to get through production's deny-all. And the platform backup with Velero: its architecture of controller, node-agent and plugins; the two routes for volumes, with the decisive difference that only the copy to the object store really gets the data out of the storage system; the pre and post hooks with pg_backup_start/pg_backup_stop that solve the consistency we left open in 05-05, and the warning that the post must always run on pain of filling the disk with WAL; the Schedules with TTLs that make retention automatic; and restoring with --namespace-mappings to verify before touching anything real.

And you have done what separates a backup from a useful backup: the drill. Deleting the whole of rutas-norte-pre and recovering it against the clock, measuring a real RTO and discovering the list of what does not come back on its own — the certificate, the LoadBalancer IP, the DNS, the external secrets, the cluster resources —, which is literally impossible to know without running it. Out of that comes the runbook someone on call can follow at three in the morning. Alongside all of that stands the warning that is not technical: the backups contain personal customer data, they must be encrypted, have a deletion period, not leave the intended legal scope, and the retention policy and the handling of that data must be reviewed and approved by the compliance officer before any of this goes into production.

This is the end of module 5. The debt we had been dragging since module 2 is settled in full: bookings-postgres keeps its data on a volume that survives the pod, dynamically provisioned by a StorageClass with Retain, expandable on the fly, with snapshots before every risky change and with verified backups inside and outside the cluster.

One ceiling remains, one we have hit three times and always postponed with the same phrase. In 05-03 you discovered that a Deployment has a single template, so all its replicas ask for the same PVC and that is why bookings-postgres is condemned to replicas: 1 with strategy: Recreate. A production database needs more: a stable identity, ordered start-up and its own volume per replica. That is what the volumeClaimTemplate of StatefulSets provides, and with it module 6, Advanced Concepts, begins: StatefulSets, DaemonSets, Jobs and CronJobs — where occupancy-reports, the platform's sixth component, finally arrives —, init containers and sidecars, scheduling with affinity and taints, custom resources and operators. We start with StatefulSets.