daily-report.sh already satisfies the seven properties of an unattended job. We are going to apply that technique to the operation you are most grateful to have automated and that goes worst when improvised. A backup has one uncomfortable quirk: it is the only job on the server whose value is only tested on the day everything else has failed. Until that day, a broken backup and a good one look very much alike — both produce files, both take up disk, both write "ok" in the log. This lesson is about what to copy, how to copy it without burning a whole disk every day, how to retain the copies and, above all, the part almost nobody tests until they need it: the restore.

1. What to back up, the 3-2-1 rule and RPO/RTO
2. Full, incremental and differential
3. tar: the compressed full copy
4. rsync: the central tool
5. --link-dest: incrementals that restore like full copies
6. Rotation and retention
7. Verification and restore
8. Databases, encryption and disk space
9. Application: backup.sh is born

The first mistake in a backup is wanting to copy everything. The whole system produces enormous files, slow to generate and to restore, and mostly reproducible: if the server burns down, you are not going to restore /usr/bin, you are going to install Ubuntu 24.04 again. What you have to back up are the three categories that cannot be rebuilt:

And what is not backed up: the operating system, the packages, the caches, the temporary files, the old logs and most especially the backup directory itself (copying it inside doubles the size on every round). A practical criterion: ask yourself how long it would take you to have that file again if it disappeared. If the answer is "I download it again", it does not go into the backup; if it is "I can't", it goes in first. The 3-2-1 rule is the industry consensus and it remembers itself: 3 copies of the data (the original and two backups), on 2 different media or systems, with 1 of them off site. Each number answers a real failure mode: two copies on the same disk die with the disk; two in the same building die with the fire or with the ransomware encryption that sweeps the network. The remote copy is the one that saves you from catastrophe, and that is why in 07-06 we will send it to srv-veloz-02 over SSH.

The other two concepts are explained with two concrete questions, no jargon. RPO (Recovery Point Objective): how much data can I afford to lose? It determines how often I make the copy. RTO (Recovery Time Objective): how long can I be without service? It determines how I make it. If Veloz Envíos accepts losing at most one day of records, the RPO is 24 hours and a daily copy is enough; if the business cannot be down for more than two hours, that RTO rules out a backup format that takes three to restore. These two numbers are decided before writing the script, because they determine the entire design.

With a full copy on Sunday and incrementals from Monday to Saturday, restoring Saturday's state requires recovering seven pieces in the right order; if Wednesday's is missing, the chain breaks. With differentials two pieces are enough, at the price of each one being bigger than the previous. The classic strategy is a weekly full plus daily incrementals, but there is a fourth option that today is the best one for files and that we will see in section 5: rsync --link-dest gives copies that are generated like incrementals and restore like full ones.

Picking up 05-01, tar packs a directory tree into a single compressible file:

c creates, z compresses with gzip, f gives the name. The -C /srv/veloz/data is important: it changes directory before packing, so inside the archive the paths are archive/2026/08/… and not srv/veloz/data/archive/…. Relative paths inside the archive means being able to restore wherever you want; with absolute paths you risk overwriting the original when extracting. The name with a date is mandatory in order to be able to rotate, and always in %F format, which sorts alphabetically the same as chronologically — something 03-08-2026 does not do.

The exclusions go in a version-controlled file, one pattern per line (*.tmp, cache/, logs/*.gz), not embedded in the script: that way they change without touching code. For large archives, --zstd compresses similarly to gzip but much faster; -j (bzip2) and -J (xz) compress more at the cost of CPU.

tar is for freezing a tree into a file. To synchronize a directory with its copy, the tool is rsync, and its virtue is that it transfers only what has changed.

The trailing slash decides the meaning, and it is the classic mistake: rsync -a /srv/veloz/data/ /backups/data/ copies the contents of data, whereas without the source's slash it creates /backups/data/data/. Always put it and be consistent. And --delete deserves a serious warning. It makes the destination an exact mirror of the source, and that means an accidental deletion in the source propagates to the backup on the next run: if somebody deletes shipments.csv and the backup runs at 03:00, your copy loses it too. Two rules: --delete only makes sense when there are several retained copies (section 6), so that yesterday's keeps what was deleted today; and you never roll it out without seeing it first.

If that list of deletions surprises you, you are not ready yet to remove the --dry-run.

This is the central trick of modern file backups. A hard link is a second name for the same data on disk: two paths, a single content, the footprint of a single copy. --link-dest DIR tells rsync: "when copying, compare each file with the one in DIR; if it has not changed, instead of copying it create a hard link".

The result is remarkable: /backups/2026-08-03/ looks like a full copy — it has all the files, it restores by copying and without rebuilding chains, and you can delete any day without affecting the others — but on disk it only takes up what changed with respect to the previous one. Each copy "weighs" 2.1 GB on its own, and the two together take up 2.2 GB because shared files are counted once. Thirty days of daily backups can take up little more than a single one.

Two warnings. Hard links require both copies to be on the same filesystem: across different disks --link-dest does not fail, it simply makes full copies and fills the disk. And since the files are shared, modifying a file inside one backup modifies it in all the ones that share it: backup directories are read-only, they are restored by copying out and they are never edited in place.

Without retention, the disk fills up and the backup stops working exactly when you need it most. The usual policy is GFS (grandfather-father-son): copies with decreasing density towards the past — 7 daily (the last week, day by day), 4 weekly (the last month) and 6 monthly (the last half year). That is 17 copies instead of 180, and it still lets you recover "the way it was on August 3rd" or "the way it was in March". The reason for keeping old copies is not disk failure — yesterday's is enough for that — but silent damage: a corrupt or deleted file that nobody notices until two months later.

In the first one, -mindepth 1 stops find from considering the root directory itself, -maxdepth 1 keeps it from going inside each copy and -exec … + groups the deletions. But -mtime depends on the filesystem's date, which is fragile (a touch or a badly made copy alters it); since the names carry an ISO date, counting by name with sort -r is more reliable. And notice the safeguard on the (( )) line: it is never redundant, because the worst possible failure of a retention script is deleting everything.

A 2 GB file with the right name may be empty inside, truncated for lack of disk or corrupt. The script finishing with code 0 does not prove the copy is any good. There are three levels, from cheapest to most expensive:

tar -t lists the contents without extracting and, by walking it all, gzip validates its internal checksum: cheap, and it detects truncation and corruption. The .sha256 file lets you check much later that the archive has not degraded and that it arrived intact on another machine. Add a common-sense check that catches the most frequent failure: (( $(stat -c %s "$dest") > 1024 )), because a backup that took 2 GB yesterday and 40 KB today is not a small backup, it is an empty backup with a silent failure behind it.

The third level is the only one that proves anything. The question that reveals whether a backup system works is not "are the copies being made?", it is "when was the last time you restored one?". The test restore is always done into a temporary directory, never over live data: extracting over the original is the fastest way to turn an incident into a disaster, because if the backup was corrupt you have just destroyed what was left too.

To restore only part of it you add the internal path at the end of the tar xzf. And from a copy made with rsync, restoring is copying in the opposite direction — without --delete, so as not to remove from the destination what was created afterwards: rsync -a /backups/daily/2026-08-03/data/ /srv/veloz/data/. The procedure must be written down somewhere you can read without access to the downed server: the exact commands, in what order, which service to stop first and how to verify afterwards. A restore runbook that only exists in one person's head does not exist.

Databases. Copying /var/lib/mysql with the service running produces an inconsistent backup: while rsync walks the files, the engine is writing to several at once, and you end up with a snapshot where some parts are from 03:00:01 and others from 03:00:47. It may look like it works and fail months later on restore. The right way is to ask the engine for a consistent copy:

--single-transaction takes the snapshot inside a transaction, without blocking writes; -Fc produces a compressed format that allows restoring individual tables. The dump is then a normal file that you can compress, verify, retain and send off site. The credentials go in ~/.my.cnf or ~/.pgpass with mode 600, never on the command line, which is visible in ps to any user.

Encryption. A copy that leaves the server takes customer data outside your control. Encrypting it is one line: gpg --batch --symmetric --cipher-algo AES256 --passphrase-file "$BASE/etc/backup.key" -o "$dest.gpg" "$dest". The --batch avoids any interactive question (property 1 from 07-02). And here is the trap: if you lose the key, you lose the backup. Keeping it only on the server you are backing up is useless — it burns with it; putting it next to the encrypted backup is worse than not encrypting. The key goes to a secrets manager or a physical envelope, outside the system (08-03).

Disk space. A backup that runs out of space halfway leaves a truncated archive that looks valid. Check beforehand, with what you learned in 06-03:

df -P forces the POSIX format of one line per filesystem, preventing a long name from splitting the output in two. The 20% margin covers the variation between source and copy. If it does not fit, the right reaction is to fail before starting, not to die halfway.

The toolkit's third script copies /srv/veloz/data and ~/veloz-ops/etc with rsync --link-dest, compresses the archive with tar, verifies with sha256sum, applies retention and logs the result, satisfying the seven properties from 07-02:

Three decisions deserve a comment. umask 077 because a backup with customer data must not be readable by everybody. The --link-dest is built as an array (link=()) so that, when there is no previous copy, the option disappears completely from the command instead of being left empty. And ${ROOT:?} in the rm -rf is an essential safeguard: if ROOT were empty because of a configuration mistake, rm -rf "/${copies[i]}" would be catastrophic; with :? the script dies first.

In the crontab it goes at 03:15 — not at 03:00, so as not to coincide with everything else — wrapped in timeout 3600s and with >> logs/backup.log 2>&1. It is a working version, not the definitive one: what is missing is sending it off the server (07-06), checking that last night's backup exists (07-04) and the guided restore. Project 09-03 takes it to its complete version, with a restore menu, a report and tests.

• Backing up the backup directory. It doubles the size on every round until the disk fills. Always exclude it.
• rsync's trailing slash. With a slash it copies the contents; without a slash it creates one extra level. Check it before every --delete.
• --delete without retention. An accidental deletion in the source destroys the copy too.
• --link-dest across different disks, or editing inside a backup. Hard links do not cross filesystems (with no warning, it makes full copies and fills the disk); and where they do work, modifying a file modifies it in every copy that shares it.
• Copying database files hot. It produces an inconsistent copy that fails on restore. Use mysqldump/pg_dump.
• Not checking space beforehand, or storing the encryption key next to the backup. A backup truncated by a full disk looks valid until you need it; and the key next to the encrypted file is equivalent to not encrypting.
• Restoring over the original data "to test". If the copy is bad, you destroy what was left too. Into a temporary directory.
• Tip: put a monthly test restore on the calendar and treat it as a real task. A backup that has never been restored has a surprisingly low probability of working.

Exercise 1. Explain what each of these three commands does wrong and fix them.

Exercise 2. Write a veloz_verify_backup function that takes the path of a .tar.gz and checks three things: that it exists and exceeds a minimum size, that the archive reads end to end, and that its sha256 checksum matches the .sha256 that accompanies it. It must return different codes for each failure.

Solution 1.

The most serious failure is the second: a backup with no date in the name is a single-generation backup, and the day the process fails halfway you will have destroyed yesterday's good copy to leave a corrupt one from today.

Solution 2.

The sha256sum -c runs inside a subshell with cd because the .sha256 file stores the relative name; without that change of directory, the check would look for the archive in the current directory and would always fail. --status silences the output so that only the exit code speaks, which is what a script needs. And the absence of the .sha256 is logged as a warning but does not invalidate the backup: "not verified" is not the same as "verified and bad".

A backup starts by deciding what to copy: data, configuration and state, never the whole system nor the backup directory itself. The 3-2-1 rule — three copies, two media, one off site — and the two questions of RPO and RTO fix the frequency and the design before you write a line. Of the strategies, full is simple and expensive, incremental cheap and fragile to restore, differential a middle ground; but for files the best option is rsync --link-dest, which generates copies like incrementals and restores them like full ones thanks to hard links — on condition that they are on the same filesystem and that you never edit inside a backup. Of the tools: tar czf with -C, a name with %F and --exclude-from to freeze trees; and rsync with -a always, -z only over the network, --delete only when there are several retained copies and never without a prior --dry-run, minding the source's trailing slash. Retain with a decreasing policy (7 daily, 4 weekly, 6 monthly) by sorted name or by find -mtime, with the safeguard of not deleting if there are fewer copies than expected. Verify at three levels — minimum size, tar -tzf and sha256sum -c — because an unverified backup is not a backup, and restore for real, into a temporary directory and never over the original. For databases, mysqldump --single-transaction or pg_dump -Fc, because copying their files hot gives an inconsistent snapshot; and if the copy leaves the server, encryption with gpg and the key stored outside the system.

backup.sh already lives in the toolkit and runs at 03:15 with a lock, a space check and retention; project 09-03 will take it to its complete version. But now you have three automated jobs writing to three logs that grow forever, and one unanswered question: who finds out if one of them stops working? In 07-04 we attack both sides of that problem: logging — designing a line format, a veloz_log function with levels, logger to syslog, journalctl to query and logrotate so the files do not eat the disk — and monitoring — thresholds, alerts without noise, checks in the style of the classic plugins and warnings that fire only once. watchdog.sh is born.