Backups do not exist; only tested restores exist. Keep that sentence, because it is the one idea in this lesson you cannot afford to forget. On srv-tramontana there have been backups for weeks: backup_tramontana.sh generates them, verifies their sha256sum, leaves them on a volume of their own and a systemd timer fires it every night. All of that sounds very good and proves absolutely nothing, because nobody has ever restored a single one. A backup that has not been restored is a hypothesis, not a protection. This lesson closes the module by turning that hypothesis into a tested procedure, with an agreed RPO and RTO, generational retention, incremental backups that do not grow out of control, encryption, verification and a runbook somebody could follow even if you were on holiday.

1. What gets backed up and what does not: Tramontana's inventory
2. RPO and RTO: how much we can lose and how long we can take
3. The 3-2-1 rule and generational retention
4. Full, incremental and differential
5. Consistency: the file being written while you copy it
6. Tools: tar, rsync, dd and restic
7. System backups versus rebuilding from code
8. Encryption, custody and legal obligations
9. Verification and the restore test
10. Automating and watching the age of the last backup
11. Restoring: Tramontana's three scenarios
12. The recovery runbook

The first decision is not a technical one: it is deciding what deserves to be copied. The rule is simple: you back up what cannot be rebuilt.

Tramontana's concrete inventory, which is what is going to be copied:

Two acronyms that have to be agreed with Marta, not decided on your own:

• RPO (Recovery Point Objective): how much data we can afford to lose, measured in time. If the backup is daily at 02:30 and the server dies at 18:00, 15.5 hours of bookings are lost. Is that acceptable?
• RTO (Recovery Time Objective): how long the service can be down while it is restored.

Translated into Tramontana with real figures: about 25 bookings come in a day, with an average value of €313.70. Losing a whole day is around €7,842.50 in bookings that would have to be reconstructed by hand from the confirmation emails, with the reputational cost of ringing the guests.

That 4-hour RPO for the database is the decision that changes the design: the daily 02:30 backup does not meet it, and a dump every four hours is needed. Every improvement in RPO or RTO costs money and complexity; that is why the decision belongs to the business, with your technical advice.

The 3-2-1 rule sums up decades of disasters:

• 3 copies of the data (the original and two more).
• On 2 different media or systems.
• 1 of them off site.

Today on srv-tramontana we have one copy, on the same server, in the same building. If the disk fails in a way that takes the VG with it, or if somebody encrypts the machine, or if the office burns down, there is no copy. /srv/tramontana/backups is a convenience, not a protection. The modern variant adds a 1-0: at least one immutable or offline copy, and zero errors in the last verification, because modern ransomware seeks out and encrypts backup destinations reachable from the server.

GFS retention (Grandfather-Father-Son): not all backups are worth the same over time.

With that, restic forget --keep-daily 14 --keep-weekly 8 --keep-monthly 12 leaves exactly the history you agreed, with no improvised decisions.

The classic compromise: a weekly full plus daily incrementals. The incremental chain has a real risk that has to be said out loud: if one link is lost or corrupted, everything after it is unrecoverable. That is why you verify all of them, not just the last. Modern tools with deduplication (restic, borg) remove almost all of this dilemma: each backup behaves like a full one when restoring and takes up as much space as an incremental.

Copying a file while somebody is writing to it produces rubbish: the first half of the file is from before the change and the second half from after. In a database, that mixture is a backup that cannot be opened. Three solutions, in order of preference:

The snapshot in practice, applying what you learned in 05-04 and making use of the 5 GiB we deliberately left free in the VG:

The correct and complete sequence for Tramontana's database combines the first two: a logical dump with pg_dump (coherent by definition) and a snapshot for the uploads files, which have no dump of their own.

The .snar is the brain of the operation: if you lose it, the next "incremental" will be a full backup, and if you mix it between chains, the backup will be inconsistent. Keep it with the backups and take a separate copy of it. And inspect before extracting, as the course's convention requires:

Here the circle closes with the hard links from 02-06. --link-dest compares against a previous backup and, for each file that has not changed, creates a hard link instead of copying the data. The result: each backup looks and restores like a full backup, but on disk it takes up only what has changed.

It looks like 2.1 GiB — and when restoring it is — but it only consumes 118 MiB of new disk. Two warnings: hard links do not protect against the corruption of a block (all the copies share the same inode and therefore the same damage), and deleting the base copy breaks nothing, because the inode survives for as long as one link remains.

dd copies block by block, understanding nothing about files. It is good for cloning a whole disk or the boot sector, and it is the most dangerous command in this lesson: swapping if= and of= destroys the destination without asking. It is called "disk destroyer" for a reason.

It requires the device to be unmounted or frozen for the image to be consistent, and it copies the empty space as well. For regular data backups, it is almost never the answer.

Deduplication, encryption as standard, integrity verification and declarative retention. Complete real-world usage:

Notice the 41.882 MiB added that occupy 12.204 MiB for real: that is deduplication plus compression. borg is equivalent in capability; rsnapshot is the classic version of the --link-dest trick, useful if you prefer backups you can browse with ls without an intermediate tool.

Copying the entire system (an image or a tar of /) lets you go back to exactly the previous state, but it produces enormous backups and drags along the accumulated rubbish, including any latent problem. The modern alternative is to rebuild the server from code: a clean machine, a playbook that installs and configures everything, and on top of that only the data restored from the backup.

Tramontana is heading towards the second, and that is why the inventory includes /etc/tramontana, the systemd units and the package list: they are the input for the future Ansible playbook you will see in 07-06. In the meantime, the package list is your insurance:

An unencrypted backup is a leak waiting to happen. The original lives on a server with permissions, sudo, ACLs and systemd hardening; the copy ends up on a USB disk, in a bucket or in somebody's house, with none of those protections. And it contains exactly the same data.

Encrypt at source, before the copy leaves the server: restic and borg do it as standard, and with tar you chain gpg or age. Managing the keys — where they live, who has custody of them, how they are rotated, what happens if they are lost — is the subject of 06-05, and it is no small detail: an encrypted backup whose key has been lost is exactly as useful as having no backup at all.

Compliance warning (GDPR). bookings.csv and the database contain guests' personal data (name, contact details, dates of stay). The backups inherit every obligation of the original: encryption at rest and in transit, documented access control, a record of who accesses them, retention limited to the time necessary, and — the point almost always forgotten — the right to erasure extends to backups as well, which makes it necessary to have a written policy on how a deletion request is handled when the data sits in immutable copies. And if the backup also goes to a cloud provider, the data processor and the location of the data come into play. None of this is for you to decide: it must be defined and approved by the data protection or compliance officer, and your job is to implement it and to be able to prove it.

An unverified backup is a file of reassuring size. There are three levels, and you have to do all three:

Only the third proves anything. The first two are automated; the third is scheduled:

The periodic restore test is part of the procedure, not an extra: once a quarter, restore onto a clean VM, start the application, check that the 25 bookings are there and that the sum of the amounts comes to €7,842.50, time how long it took (does it meet the 8 h RTO?) and record the result with the date and the name of whoever did it. A restore test nobody has timed does not let you promise any RTO.

You already have the automation from 05-05: tramontana-backup.service with its timer, Persistent=true and RequiresMountsFor. What is missing is the watching, and here is the industry's most common mistake: watching that the job ran instead of watching that there is a recent, valid backup. A script that fails silently and a disabled timer produce the same symptom: nothing. The metric that really matters is the age of the last correct backup.

It is fired by its own timer at 08:00, after the backup window, and it only notifies when there is something to do: it is the principle of "silence if all is well" applied to the only metric that matters.

RPO 24 h, RTO 30 min. Luis ran an unfortunate mv at 11:40.

Step 4 is what separates a restore from an act of faith: 26 lines and €7,842.50 are the file's known figures. If they do not add up, the backup is no good and you have to go to the previous one.

RPO 0, RTO 15 min. It is the case of 3.3.0, which weighs 99.2 MiB and does not start. deploy.sh already rolls back on its own (05-05), but if the fault is detected later:

The daily backup plays no part here: the one that saves the day is the pre-deployment backup in /srv/tramontana/backups/pre-deploy/, and the relative symbolic link makes it possible to go back in a second. This is 02-06 and 05-05 working together.

RPO 4 h, RTO 8 h. The scenario everybody avoids thinking about. The complete procedure:

1. A new machine: Ubuntu Server 24.04 LTS, the same hostname, the same partitioning scheme with /var isolated (01-04).
2. Identities first, and with the same numbers, or the permissions from the restore will not add up: groupadd -g 1002 tramontana, useradd -r -u 997 -g tramontana -s /usr/sbin/nologin svc-tramontana, plus operator and luis (05-01).
3. Packages: xargs -a packages-2026-08-19.txt sudo apt install --no-install-recommends -y, with the database's hold (05-03).
4. Disks: recreate the PV, VG and LV, format and mount /srv/tramontana/backups by UUID with nofail, and mount -a before rebooting (05-04).
5. Data: restic restore <snapshot> --target /, and the database dump with psql < dump.sql.
6. Configuration and services: restore /etc/tramontana, the units, sudoers.d, logrotate.d; systemctl daemon-reload; systemctl enable --now tramontana.service tramontana-backup.timer (05-05, 05-06).
7. Verify: health_check.sh, count the 25 bookings, check the total of €7,842.50, an empty systemctl --failed, and test a real booking from start to finish.
8. Time it and write down how long it took. That number is the real RTO, and it is the one you can promise.

Notice that this procedure uses all eight lessons of the module. That is the point of Module 5.

A runbook is the document that lets somebody else carry out the recovery, in the small hours, in a hurry and without you. It must contain:

• An inventory of what is backed up, where it goes and how often.
• The agreed RPO and RTO, with who approved them and when.
• Where the encryption keys are and who has custody of them (not the key: where it is).
• The procedures for the three scenarios, with literal, copyable commands.
• Contacts: who decides, who executes, who tells the customers.
• The date and result of the last restore test, and who did it.

And the most important part: it cannot be kept only on the server you are going to lose. A printed copy, an external git repository, the company's document management system. A runbook that exists only in /opt/tramontana/HISTORY is a runbook that disappears in scenario (c), precisely when you need it.

• Never having restored. It is this lesson's mistake. Schedule the quarterly test today, with a date and a person responsible.
• The backup on the same server. A fire, a ransomware encryption or a VG failure takes the original and the copy. 3-2-1, no excuses.
• Losing the .snar file from tar --listed-incremental, or mixing it between chains: the incremental backup stops being recoverable.
• Copying a live database with cp. It produces a useless file. A native dump or a snapshot.
• Unencrypted backups leaving the server, or encrypted ones with a key nobody has custody of. Both are equally serious.
• Watching the job instead of the result. What matters is the age of the last correct backup, not that the script finished.
• Restoring straight over the original. Restore to a separate directory, verify and then put it in place.
• Infinite retention "just in case". It costs money and clashes with the GDPR: retention is agreed and documented.
• Tip: time every test restore. The RTO you can promise is the one you have measured, not the one you would like.

1. Designing the scheme. Starting from the 4 h RPO and the 8 h RTO agreed with Marta, describe Tramontana's complete backup scheme: what is copied, how often, with which tool, where each copy goes and what retention it has. Justify where the 3-2-1 rule fails today.
2. Verifying without a full restore. Write the commands that check, without touching production, that last night's backup contains bookings.csv with the 25 bookings and that the sum of the amounts is correct. Explain why this does not replace a restore test.
3. The missing link. You have a full backup from Sunday and six tar incrementals. Wednesday's is corrupt. Up to which day can you restore, and why? What would you have done differently so that this problem did not exist?

1.

Where 3-2-1 fails today: there is one single copy (the "3" fails), on the same server and the same logical disk (the "2" fails) and with none off site (the "1" fails). The acceptable minimum is to add a replica of the restic repository to a remote destination with append-only credentials, so that a compromise of the server does not allow the history to be deleted.

2.

restic dump extracts a file to stdout without writing anything to disk, so the check does not touch production and needs no space.

Why it does not replace a restore test: this verifies one file, not the whole set; it does not check permissions, owners or ACLs; it does not validate that the restored database starts or that the application works with that data; and, above all, it times nothing, so it does not let you claim that the RTO is met. Verifying is necessary; restoring is what proves it.

3. You can restore up to Tuesday: Sunday's full backup plus Monday's and Tuesday's incrementals. Wednesday's is corrupt, and since each tar --listed-incremental increment contains only what has changed since the previous one, Thursday's, Friday's and Saturday's depend on a state you can no longer reconstruct. A file modified on Wednesday and never touched again exists only in the broken link.

What would have avoided the problem, in order of effectiveness:

• Verifying every backup as it is created (tar -tzf and sha256sum), not just the last one: on Wednesday you would have known it had to be repeated.
• Using differentials instead of incrementals: each one depends only on the full backup, so a broken link costs one day, not four.
• Better still, using a tool with deduplication and verification such as restic, where each snapshot restores on its own and restic check --read-data detects corruption before you need it.

You have closed Module 5, and with it the distance between "I have scripts" and "I run a server". Look back at what has changed on srv-tramontana since you started:

• 05-01 — Identities stopped being magic: the tramontana group (gid 1002) and the svc-tramontana service account (uid 997, with no shell and no password) genuinely exist, with operator and luis in the group and ownership of /opt/tramontana and /srv/tramontana/backups in its place.
• 05-02 — sudo stopped being a magic word: there is a written, validated, documented rule in /etc/sudoers.d/tramontana, plus SGID on the shared directories, ACLs so that Luis reads the logs without joining adm, capabilities instead of SUID and chattr +i on app.conf.
• 05-03 — The software has provenance: deb822 repositories with keys in /etc/apt/keyrings/, the database pinned with hold and pinning, and unattended security updates that warn without rebooting on their own.
• 05-04 — The backups live on their own LVM volume that grows while in use, mounted by UUID with nofail, with margin in the VG for snapshots and an fstab tested with mount -a.
• 05-05 — The application is a real service: a hardened tramontana.service, with automatic restart and cgroup limits, and the nightly backup turned into tramontana-backup.service + .timer with Persistent=true; deploy.sh finally restarts.
• 05-06 — The server has a memory: a persistent, capped journal, /etc/logrotate.d/tramontana tested in simulation, the scripts writing to the journal, and the exact answer to "what happened last night at three?".
• 05-07 — And it has a thermometer: a baseline with thresholds, sysstat keeping history, the USE method and an eleven-step procedure that has already solved the morning slowness by measuring before and after.
• 05-08 — And now it has a safety net: an RPO and RTO agreed with Marta, the 3-2-1 rule, GFS retention, consistency through a dump and an LVM snapshot, restic with deduplication and encryption, verification at three levels, watching the age of the last correct backup and three restore procedures written, tested and timed.

None of this was "knowing commands". It was building a system that stands up when you are not in front of it, and that is exactly the difference between using Linux and administering it.

And yet everything you have built in this module has a door wide open. srv-tramontana listens on 8080 with no firewall, the network configuration is still the one the installer left, SSH accepts passwords and root logins without anybody having reviewed sshd_config, the db_password sits in the clear inside app.conf, the booking traffic — with the guests' personal data you have taken such care to protect in the backups — travels unencrypted, and in /var/log/btmp there are already forty-seven login attempts from an IP nobody has blocked. In Module 6: Networking and Security you close that door: you will configure the network persistently with netplan, you will harden SSH with keys and no root, you will raise a firewall that only lets through what is essential, you will detect and stop intrusions, you will get the secrets out of the configuration files and put TLS in front of the application, and you will finish by applying a complete hardening with AppArmor and a password policy that today is still the factory one. You have made a server that works; now it is time to make it defensible. Update your VM snapshot, keep the runbook off the machine and I will see you in Module 6.