/srv/tramontana/backups has been growing for weeks. Every night at 04:20, backup_tramontana.sh leaves a new backup there without anybody having worked out how much fits, and the 23 GB root disk is at 30%. The question is not whether it will fill up, but when — and what will happen to the application when it does, because a full / is not "it is running slowly": it is a machine that cannot even write a log or complete a transaction. In this lesson you finally go down to the hardware: you will see the full stack from the physical disk to the mount point, you will partition, format, understand /etc/fstab field by field (and why one badly written line stops the machine booting), and set up LVM, which is what a real server uses. By the end, /srv/tramontana/backups will live on its own logical volume, expandable while it is in use.

1. The storage stack, from top to bottom
2. Identifying the hardware: lsblk, blkid, fdisk -l
3. Partition tables: MBR versus GPT
4. Partitioning with fdisk and parted
5. Filesystems compared, and inodes revisited
6. Mounting: mount, options, bind mounts and findmnt
7. /etc/fstab field by field and the safety net
8. Space: df, du, and the deleted file that does not free it
9. Swap: partition, file and how much you need
10. LVM: PV, VG, LV, growing live and snapshots
11. Software RAID with mdadm
12. Disk quotas
13. Tramontana case: a new disk for the backups

Between the platter (or the NAND cell) and the cat file.txt you type there are five or six layers. Confusing them is the cause of 90% of storage mistakes.

The LVM layers are optional — without them the partition is formatted directly — but they solve the worst problem of all: a partition cannot be enlarged unless the free space is right behind it; a logical volume can, even if the space is on another disk.

lsblk -f is the command you will use most often: it tells you the filesystem type, the label, the UUID and the actual usage.

And the critical detail: these names can change between boots. It only takes adding a disk, changing the order of the controllers or the kernel detecting devices in a different order for today's sdb to be tomorrow's sdc. That is why you never mount by device name in fstab: you mount by UUID or by LABEL, which travel inside the filesystem itself.

For any new disk today: GPT, with no exceptions worth discussing. The 1 MiB partition you see as sda1 is precisely the bios_grub one GRUB needs on a GPT disk booted by legacy BIOS; its role is explained in 07-01.

fdisk is interactive and convenient; parted accepts one-line commands, which makes it suitable for scripts. We prepare /dev/sdb with a single partition taking up the whole disk:

The 1MiB start is not a whim: it aligns the partition with the physical blocks and avoids a noticeable performance penalty on SSDs and on storage arrays.

In fdisk the equivalent session would be g (create a GPT), n (new partition), Enter three times, t and 31 (type Linux LVM) and w to write. Nothing is written until the w: q gets you out without touching anything, and that is fdisk's safety net.

If the kernel does not notice the new table — typical when the disk has a partition mounted — sudo partprobe /dev/sdb re-reads it, and lsblk /dev/sdb confirms that sdb1 has appeared.

A practical rule for a conventional Ubuntu server: ext4 on LVM. You get resizing, snapshots (LVM's) and the most battle-tested filesystem there is.

tune2fs lets you change the label (-L), the check interval (-i) and, very useful on a data volume, the 5% reserved for root: sudo tune2fs -m 1 /dev/vg-data/lv-backups recovers about 600 MiB out of 15 GiB. That 5% makes sense on / — it stops a full disk preventing root from fixing the situation — but on a volume dedicated to backups it is wasted space.

In 02-06 you saw that the inode holds the metadata and that the name is just a directory entry. Here comes the operational consequence: the number of inodes is fixed when you format and it does not grow. A directory with millions of tiny files can exhaust them with the disk half empty, and the error is baffling:

43% of free space and No space left on device when creating a file. Faced with that error, df -i is the mandatory second check. The solution means deleting small files or reformatting with more inodes (mkfs.ext4 -i 8192).

If umount answers "target is busy", pick up lsof and fuser from 03-06:

Somebody — you, probably — has their working directory there. umount -l (lazy) unmounts when it is released, but it is a patch.

findmnt is infinitely more readable than mount with no arguments, and findmnt --verify validates the fstab without mounting anything. A bind mount (sudo mount --bind /srv/tramontana/backups/outgoing /opt/tramontana/output) makes an existing directory appear at another point in the tree without copying anything: it is the clean way of exposing a folder to a confined service, and in 05-05 you will see that systemd does the same thing with BindPaths.

At boot, systemd turns each line of fstab into a .mount unit and makes them a dependency of local-fs.target. If a device does not turn up, the boot waits 90 seconds and then drops into emergency mode asking for the root password — which on Ubuntu is locked. A remote server in that state is a lost server until somebody opens the console.

That is why there are two non-negotiable rules:

mount -a before rebooting is not optional. If it fails, you fix it with the server on its feet; if you do not run it, you will find out with the server down. And add nofail to everything that is not /: it turns a boot failure into a warning in the log.

ncdu does the same thing interactively and is what you will use when you are hunting the culprit in a hurry. Remember from 02-03 that du measures space occupied in blocks and ls -l the logical size: they do not have to match.

Gigabytes are missing that nobody can see. The explanation connects directly with the inodes from 02-06: when you delete a file that a process has open, the name disappears from the directory, but the inode and its blocks stay alive until the last descriptor is closed. Somebody ran rm on a huge log and the process is still writing into a file with no name.

NLINK 0 and (deleted) are the signature of the problem. The two ways out: restart the process (systemctl restart, which we will see in 05-05) or, if you cannot, truncate the file through its descriptor with sudo truncate -s 0 /proc/1284/fd/3, which frees the space without killing it.

And the underlying lesson: a log is not deleted, it is rotated — which is exactly what we will set up in 05-06.

Swap is disk space the kernel uses to offload rarely used memory pages. Nowadays a swap file is preferred to a partition: it is resized without touching the partition table.

With its line in fstab (/swapfile none swap sw 0 0). The chmod 600 is mandatory: the contents of swap include chunks of processes' memory, and that includes secrets.

vm.swappiness (0–200, 60 by default) decides how eager the kernel is to use it; lowering it to 10 is the usual choice on an application server. Fine-tuning this and other kernel parameters is the subject of 07-03. How much swap? With 3.8 GB of RAM, between 2 and 4 GB is reasonable: not as a substitute for memory — if the server swaps constantly it is going to be very slow anyway — but as a cushion that stops the OOM killer from killing the application during an occasional peak.

LVM introduces three levels: the PVs (partitions or whole disks handed over to LVM), the VG (a pool of space made up of one or more PVs) and the LVs (the "virtual disks" that get formatted and mounted). The VG is divided into 4 MiB extents, and an LV is simply a set of extents, which do not have to be contiguous or on the same disk. That is where all of LVM's power comes from.

Notice that we have deliberately left 5 GiB free in the VG: it is the margin for growing the volume and, above all, for being able to create a snapshot. A VG at 100% is a VG with no room to manoeuvre.

Growing it live, with the volume mounted and in use:

The -r option (--resizefs) is the key one: it grows the LV and the filesystem on top of it in a single step. Without it you would have a bigger volume and an identical df, which is the most frequent mistake of anybody starting out with LVM. lvextend -l +100%FREE -r eats all the VG's free space.

Shrinking is another story. You have to do it the other way round (the filesystem first, then the LV), with the volume unmounted, and XFS simply cannot be shrunk. Getting the order wrong here destroys data. The rule: create LVs small and grow them when you need to; never the other way round.

A snapshot is an LV that stores the differences from the original since the moment it was created. It is created in seconds and lets you copy a volume in a frozen, consistent state while the application keeps writing:

Two indispensable warnings: the snapshot fills up if the original changes more than its size allows, and when it fills up it is invalidated and lost. And it is not a backup: it lives in the same VG, on the same disk. It is a consistency tool, and that is how we will use it in 05-08.

[UU] means both disks are healthy; a [U_] is a failed disk and you have to act today, not next week. An LVM PV is then put on top of /dev/md0, and that is how redundancy and flexibility are combined.

RAID is not a backup. RAID protects you against a disk breaking. It does not protect you against an rm -rf (which is replicated to the mirror instantly), nor against ransomware encryption, nor against a controller fault that corrupts both disks, nor against a fire in the room. The backup is 05-08, and they are different, complementary things.

When several users share a volume, quotas limit how much each of them takes up. They are enabled with usrquota,grpquota in fstab, initialised with quotacheck -cug and set with edquota -u user or non-interactively: sudo setquota -u luis 5G 6G 0 0 /srv/tramontana/backups (soft 5G, hard 6G), and sudo repquota -s /srv/tramontana/backups gives the usage report. The soft limit can be exceeded during a grace period; the hard one is never exceeded. On an application server with a single service user they are of little use; on a shared one they are the difference between an incident and a phone call at three in the morning.

Marta approves adding a 20 GiB disk to the VM. The goal: to stop /srv/tramontana/backups competing for /'s space, without losing a single byte of the existing backups and without backup_tramontana.sh noticing.

Step 1 — Add the disk. With the VM powered off, VirtualBox → Storage → a 20 GiB disk. On booting, lsblk should show sdb with no partitions.

Step 2 — Partition, PV, VG and LV: the commands from sections 4 and 10, already run. The result: a 20 GiB vg-data with a 15 GiB lv-backups and 5 GiB of margin.

Step 3 — Format and mount at a temporary point.

Step 4 — Copy preserving everything. rsync -aHAX is not a luxury: -a preserves permissions and owners (the operator:tramontana from 05-01), -H the hard links, -A the ACLs from 05-02 and -X the extended attributes.

(Before copying, comment out the backup's cron line so that one does not start halfway through the job.)

Step 5 — Swap over, with a safety net. We do not delete the original: we rename it, so the rollback is instantaneous. sudo umount /mnt/new && sudo mv /srv/tramontana/backups /srv/tramontana/backups.old && sudo mkdir /srv/tramontana/backups.

Step 6 — fstab, with the three protections.

nosuid and nodev because on a backup volume there must never be a SUID binary or a device file; nofail because a fault on this disk cannot be allowed to stop the server booting.

Step 7 — Restore ownership, verify and clean up.

Only when a real backup has worked and its sha256sum has been verified do you delete /srv/tramontana/backups.old, and you write everything down:

• Mounting by /dev/sdb1 in fstab. The day the detection order changes, the wrong disk will boot, or none will. UUID or LABEL, always.
• Rebooting without mount -a. It is the number one cause of servers that do not come back. And nofail on everything that is not /.
• lvextend without -r. The volume grows, df does not change and you lose half an hour. With -r, the filesystem grows with it. And do not fill the VG to 100%: with no free space there is no snapshot and no margin to grow.
• Trusting RAID as a backup. It is not one, and finding out on the day of the accidental deletion is expensive.
• rm on a log that is in use. The space is not freed until the process closes the descriptor: lsof +L1 gives it away. Logs are rotated (05-06).
• Copying data with cp -r instead of rsync -aHAX: you lose owners, hard links, ACLs and attributes, and then the permissions from 05-01 do not add up.
• Tip: keep lsblk -f, vgs, lvs and a copy of the fstab alongside the HISTORY. Reconstructing the disk layout from memory, with the server down, is not a plan.

1. Diagnosing a full disk. df -h / shows 100%, but du -sh / adds up to considerably less. List the three possible causes in order of likelihood and the command that confirms each one.
2. Growing it live. lv-backups (15 GiB) has run short and the VG has 5 GiB free. Grow it by 3 GiB without unmounting, verify that the space is visible to df and explain what would have happened without -r.
3. A safe fstab entry. Write the fstab line for a user-uploads volume mounted at /opt/tramontana/shared/uploads, with the options a directory written to by third parties must have, and describe how you would validate it before rebooting.

1. In order of likelihood:

The third is the most treacherous: if somebody wrote into /srv/tramontana/backups before the volume was mounted on top, those files still take up space on / but are invisible because the mount covers them. Bind-mounting / at another point brings them into view. It is the risk in step 5 of the Tramontana case, and that is why we created an empty directory instead of reusing the one that had data.

2.

Without -r, lvs would show 18 GiB and df would go on saying 15 GiB: the ext4 filesystem does not know that the device underneath has grown until it is told with resize2fs. The operation is safe while mounted because ext4 can grow mounted; shrinking would require unmounting and doing it in the reverse order.

3.

noexec is the key addition compared with the backup volume: in a directory written to by third parties, preventing binaries from running stops a malicious upload turning into running code dead in its tracks. nodev and nosuid close the other two routes, and nofail stops a problem with this volume preventing the boot. Validation before rebooting:

Note that findmnt at the end shows the effective options: it is the only proof that what you wrote is what the kernel applied.

There is no longer an opaque layer between your files and the hardware. You walk the whole stack — disk, partition table, partition, PV, VG, LV, filesystem, mount point — and you know what problem each rung solves; you identify the hardware with lsblk -f, blkid and fdisk -l, and you know why the /dev/sdX names are not trustworthy and the UUID is; you choose GPT without hesitating, you partition with parted in one line and with fdisk knowing that nothing is written until the w; you compare ext4, XFS, Btrfs and ZFS with judgement, you tune with tune2fs and you recognise the bewilderment of a df -h with room and a df -i at 100%.

You mount with the options a server needs — noatime, nodev, nosuid, noexec, nofail —, you read /etc/fstab field by field and you know that mount -a before rebooting is the difference between a tweak and a lost night. You diagnose a full disk in its three variants, including the deleted file that lsof +L1 gives away. And you have mastered LVM: you create PVs, VGs and LVs, you grow them live with lvextend -r, you know why shrinking is dangerous and you use snapshots to freeze a consistent state. You know mdadm and its levels, and you have it engraved that RAID is not a backup.

Above all, /srv/tramontana/backups now lives on its own 15 GiB logical volume, expandable while in use, mounted by UUID with nofail, with the backups migrated byte by byte with rsync -aHAX and verified, and with 5 GiB of margin in the VG reserved for snapshots.

And now we reach the heart of the module. You have identities, permissions, packages and disk, but the Tramontana application is still started by hand: if the server reboots, it does not come back; if the process dies, nobody brings it up; deploy.sh changes the symbolic link and still restarts nothing; and the 04:20 backup depends on a cron line with no real overlap control and no decent logging. In systemd and Service Management all of that ends: you will write tramontana.service from scratch and hardened, you will understand the real difference between ordering and dependency, you will turn backup_tramontana.sh into a .service with its .timer and Persistent=true, and deploy.sh will finally do the systemctl restart it was missing.