We ended the previous lesson pointing at a seam we have been ignoring all module long. When we walk /var/lib/meteora/readings/2026-08-31.dat component by component, we take for granted that the tree is a homogeneous thing. It is not. / is an ext4 on a partition of the first NVMe; /var/lib/meteora is another ext4, on the RAID 1 /dev/md0; /run is tmpfs and lives in RAM; /proc has no device behind it at all and its files are invented at the moment you read them. Four radically different file systems, with incompatible internal structures, and yet path resolution walks from one to the next without noticing.

This lesson explains how that is achieved, and it does so in two complementary halves. The first is administrative: how a block device is carved into partitions, how LVM adds a layer of flexibility on top, how a file system is created and — the central operation — what exactly happens when you run mount, with /etc/fstab field by field and the options that really protect you. The second is architectural: the Virtual File System (VFS), the kernel's indirection layer that makes the same open() work on an SSD, on RAM and on a server at the other end of the network.

By the end you will know how to partition and mount a server with judgement, understand why separating /var from / is not a fad but prevention, resolve the "target is busy" everybody suffers, and explain precisely why /proc/cpuinfo behaves like a file without being one.

1. From block device to partition: MBR and GPT
2. A server's partitioning scheme and why to separate /var
3. Seeing what is there: lsblk, fdisk -l and blkid
4. LVM: physical volumes, volume groups and logical volumes
5. LVM snapshots for backing up /var/lib/meteora live
6. Creating the file system with mkfs
7. Mounting: what exactly happens when you run mount
8. Stable identification: UUIDs, labels and /etc/fstab field by field
9. Mount options and what each one protects against
10. Unmounting, "target is busy" and how to resolve it
11. Bind mounts and mount namespaces
12. The Virtual File System (VFS) and its four objects
13. Virtual and in-memory file systems
14. Network file systems: NFS and SMB

In module 2 we left the disk as a vector of LBA-addressable blocks. A partition is simply a contiguous range of that vector declared as an independent unit: "from LBA 2048 to 1050623, this is one thing". Nothing more. The partition imposes neither format nor content; it only delimits.

Why partition instead of using the whole disk? For four reasons that are still valid: isolating the filling (if /var/log overflows, that must not stop you from logging in), applying different policies to each area at mount time, using different file systems according to the workload (04-01), and satisfying the boot requirements (UEFI demands a FAT32 partition; disk encryption needs an unencrypted /boot).

The partition table lives in the disk's first sectors, and there are two formats. MBR (Master Boot Record), from 1983, occupies the first 512 bytes: 446 for the boot code, 64 for the table — 4 entries of 16 bytes — and 2 for the 0x55AA signature. GPT (GUID Partition Table), part of UEFI, uses a header at LBA 1 and an array of 128-byte entries, with a complete copy at the end of the disk.

Three differences really matter. MBR's 2 TiB limit is insurmountable: start and size are stored in 32 bits of 512-byte sectors, and 2³² × 512 = 2 TiB, so any 4 or 8 TB disk requires GPT. Redundancy: with MBR, corrupting 512 bytes destroys the table and access to everything; GPT keeps a full copy at the end and verifies both with CRC32, so that gdisk rebuilds one from the other — the difference between "I have lost the disk" and "that took me a minute". And the protective MBR: GPT writes a fake MBR at LBA 0 with a partition of type 0xEE covering the whole disk, so that an old tool sees it as occupied and does not cheerfully format it.

Current rule: always use GPT, unless you need to boot a very old legacy BIOS.

This is the real scheme of meteo-01, with two 512 GB NVMe drives in RAID 1 (02-05) plus the data volume:

And now the justification, which is what matters. Separating /var from / is not an inherited ritual: it prevents a concrete and frequent failure mode.

/var contains everything that grows without control: logs, mail queues, caches, databases, container images. If it shares a partition with / and a log runs away, no process can create temporary files, systemd cannot write its state, sudo may fail because it cannot log, the shell keeps no history and no lock files, and — the serious part — you cannot log in, because your profile needs to write.

That last consequence is the whole argument: a full / is a server that will not let you in to fix it. With /var separate, a runaway log fills /var, the services on that partition fail, and / keeps enough space for you to connect, diagnose and delete. It is also the reason for the -m 5 that ext4 reserves for root by default (04-01) and why you must not lower it on /.

The same reasoning, applied to the rest: /boot keeps the kernel reachable even if / is encrypted or damaged; /home stops a user from filling the system and allows reinstalling without losing data; /tmp mounted noexec,nosuid cuts off many exploits; and /var/lib/meteora isolates unrecoverable data on its own RAID, with its own mkfs and its noatime.

The honest counterargument: fixed partitions waste space — / with 30 GiB free does not help a full /var — and they are hard to resize. That is exactly the shortcoming LVM comes to solve.

Three tools, three points of view. Never format or partition without having looked at all three.

lsblk shows the tree of block devices: what contains what.

The valuable part is the hierarchy: you can see that md0 is made of two partitions on different disks and that it is md0, not they, that carries the ext4. TYPE distinguishes disk, part, raid1, lvm and crypt. And an empty MOUNTPOINT field means it is not mounted, which is the first thing to check before touching anything.

fdisk -l goes into the partition table:

What it adds that lsblk does not: Disklabel type: gpt (it confirms the format), the exact LBAs of start and end, and the logical/physical sector size. Starting at sector 2048 is no accident: it aligns the first partition to 1 MiB, which guarantees that the file system's 4 KiB blocks line up with the disk's physical sectors and with the SSD's erase pages. A badly aligned partition causes read-modify-write cycles on every operation and can cost 30 % of the performance; today the tools align on their own, but it is worth knowing why that 2048 is there.

blkid identifies the content: which file system is there and with which UUID.

There is the UUID we will use in /etc/fstab, and the meteora-data label we set with mkfs -L in 04-01. Mind the distinction: UUID identifies the file system (written by mkfs into the superblock) and PARTUUID identifies the partition (written by the GPT table). Reformatting changes the UUID but not the PARTUUID.

The problem with fixed partitions is that you decide the sizes on installation day, when you know the least, and changing them afterwards is risky. LVM (Logical Volume Manager) inserts an indirection layer that solves it, with three levels:

• PV (physical volume): a block device — partition, whole disk or RAID — handed over to LVM.
• VG (volume group): the union of several PVs into a single pool of space, divided into 4 MiB extents.
• LV (logical volume): a virtual block device built from the VG's extents. You run mkfs on it and mount it, exactly as on a partition.

The full creation, and then extending it live, without unmounting or stopping the service:

Two steps because there are two layers: lvextend enlarges the virtual device and resize2fs extends the ext4 so it takes up the new space. Order matters: when extending, the volume first and the file system second; when shrinking, the other way round — resize2fs first to shrink the ext4 and then lvreduce — because if you shrink the volume before the file system, you cut through data and lose it. lvresize -r performs both steps in the right order and is the safe way to do it. A note that connects with 04-01: ext4 can be shrunk; XFS cannot, because xfs_growfs only grows; it is a strong argument in favor of ext4 when you do not know how your volumes will evolve.

Here is the feature that solves a real Meteora problem. Copying /var/lib/meteora/readings/2026-09-01.dat while the ingestor is writing to it produces an inconsistent copy: the beginning of the file is from 03:00 and the end from 03:04, with a Reading struct possibly cut in half. Stopping the service for half an hour every night is not acceptable.

An LVM snapshot freezes a volume's state at an instant:

How it works: the snapshot copies nothing when it is created. It is an empty table plus a copy-on-first-write rule (copy-on-write): when the ingestor writes to a block of the original volume, LVM first copies the old content into the snapshot's area and then lets the write through. This way, whoever reads the snapshot sees the frozen state and whoever reads the original sees the current one.

And now the warning, which is the part people ignore and later regret:

Sizing it is arithmetic: if Meteora writes 17.3 MB a day and the backup takes 20 minutes, about 240 KB will change, so 20 GB leaves ample margin even for an anomalous spike. Monitoring is done with lvs, watching the Data% column; if it approaches 100 %, you have to extend it with lvextend now:

A point of technical honesty: the snapshot freezes the device's blocks, not memory. If the ingestor had data in the page cache not yet flushed, the snapshot does not have it. That is why a perfect backup takes one preliminary step: ask the service to fsync() (or send it SIGHUP), run sync, and only then create the snapshot. We will see this when we talk about fsync in File Management, and consistency guarantees are the subject of Space Allocation, Journaling and Integrity.

mkfs writes onto the device the structures we studied in 04-01: superblock, group descriptors, bitmaps and inode table. It does not ask and it does not warn: if the device is the wrong one, the previous data becomes inaccessible instantly.

The options that are really used, and what each one decides:

The safe procedure, dry run included:

The first three commands answer "is this the right device?", "what does it hold now?" and "is it mounted?" in five seconds, and they prevent the most expensive accident in systems administration. Step 4 with -n prints exactly what it would do — number of inodes, of blocks, positions of the backup superblocks — without touching anything.

We reach the central operation. Mounting is grafting a file system's tree onto a point of the global tree, so that path resolution crosses from one to the other without noticing.

The steps the kernel performs, in order:

1. Resolve the mount point. /var/lib/meteora is turned into an inode, with the algorithm of 04-02. It must exist and be a directory; otherwise, ENOTDIR or ENOENT.
2. Open the device and read its superblock, which in ext4 starts at byte 1024: from it come the block size, the number of inodes, the position of the inode table, the UUID, the enabled features and the state.
3. Check the state. If it says "dirty" — it was not unmounted properly — recovery from the journal is triggered (04-05). If it requires features this kernel does not support, the mount is rejected.
4. Create a superblock object in memory, the live representation of the file system, with its operations (read inode, write inode, statistics...).
5. Create a vfsmount structure associating that superblock with the mount point's inode.
6. Hook the dentry. From now on, when path resolution reaches the dentry of /var/lib/meteora, it will see the "there is a mount here" mark and jump to the root inode of the new file system.
7. Register it in the mount table, visible in /proc/mounts.

Step 6 is the key to everything, and it explains mounting's most baffling behavior:

Mounting over a non-empty directory does not delete its content: it hides it. The files are still there, intact, but the path no longer reaches them because step 6 diverts resolution.

It is easily verified:

Hence a classic incident: somebody writes data into /var/lib/meteora before mounting the volume, then mounts, and that data disappears from view while still taking up invisible space on /. To see it without unmounting there is the bind mount trick of section 11.

The mount table:

/proc/mounts is the kernel's truth, with the options actually in force. mount with no arguments shows the same thing, and findmnt presents it as a tree: it accepts a mount point or a device, filters by type (-t ext4) and returns an exit code that is useful in scripts, so it is the tool to use.

A real problem: the /dev/sdX names are not stable. They are assigned in the order the kernel detects the disks, which depends on how fast each controller comes up, on the order of the ports and on whether a USB drive is plugged in; the disk that is /dev/sdb today may be /dev/sdc tomorrow, and an fstab entry naming it will mount the wrong volume or fail the boot. The four ways of identifying a device:

Use UUID= in /etc/fstab. It is what every modern installer does. The only precaution: the UUID changes when you reformat, so after a mkfs you have to update fstab or the system will not boot.

The real /etc/fstab of meteo-01:

The six fields, one by one:

1. Source. What to mount: UUID, label, device, or an arbitrary name for the virtual systems (tmpfs, proc).
2. Mount point. Where. For swap, none.
3. Type. ext4, vfat, tmpfs, swap, nfs... or auto so that mount deduces it by reading the superblock.
4. Options, comma-separated and with no spaces. It is the field covered in the next section.
5. dump. A relic of the dump utility from the 1980s. Today always 0.
6. pass. The fsck checking order at boot: 1 for the root, 2 for the rest (they are checked in parallel if they are on different disks) and 0 for no check — mandatory on tmpfs and network systems, which have no fsck.

The most dangerous mistake in this whole module lives in fstab: a badly written line can stop the machine from booting, leaving it in emergency mode with no remote access. The safe procedure is non-negotiable:

findmnt --verify checks that the UUIDs and mount points exist and that the options are valid without mounting anything. mount -a tries to mount everything: if there is an error, it appears now, with the machine alive, and not at boot. And if something goes wrong anyway, the nofail option prevents one entry's failure from blocking the boot — highly recommended on external disks and network mounts.

The fourth field's options are one of the cheapest hardening tools there is: they cost zero performance and close entire classes of attack.

The trio noexec,nosuid,nodev is the standard recipe for any volume containing only data. nosuid is the most important: without it, an attacker with write permission can drop a copy of /bin/bash with root's setuid bit and get a superuser shell; with nosuid the kernel ignores that bit and the attack does not work — one word in fstab against a root escalation. nodev closes the analogous route with devices: without it you could create a device file with the major/minor of /dev/sda and read the whole disk bypassing permissions. And noexec stops binaries from being executed from there, although it is not a strong barrier: a script still runs with bash script.sh and a binary with /lib/ld-linux.so.2 ./binary, so use it as a layer of defense, not as a wall.

That is why /var/lib/meteora is mounted with noatime,nosuid,nodev,noexec: it contains data, never programs, so all four options are free and close three vectors. Applying the same trio to /tmp, /var and /home is one of the best effort/benefit ratios in systems administration, and we will come back to it in module 5. The options can be changed live with mount -o remount,ro /var/lib/meteora (and rw to go back), which is exactly what the kernel does with errors=remount-ro when it detects an I/O error: freeze the writes so as not to make the damage worse.

umount unmounts: it flushes pending writes, syncs the superblock, marks it as clean (decisive for 04-05) and unhooks the mount point. But:

A file system is busy if some process is using it, and "using" includes four cases people forget:

1. It has a file open inside it (04-04).
2. It has its cwd inside it (04-02).
3. It has a file mapped into memory with mmap() (02-04).
4. There is another mount on top of one of its subdirectories.

The diagnosis, with two complementary tools:

In lsof, the FD column is the key: cwd is the working directory, mem is a mapped file, and a number (7r) is an open descriptor. fuser -vm /var/lib/meteora says the same thing with letters: c (cwd), e (executable in use), f (open file), r (root) and m (mapped).

The solutions, in order of preference:

About the last three, honestly. umount -l (lazy) unhooks the mount point immediately from the hierarchy, but keeps the file system alive until the last descriptor is closed: the paths stop working instantly, but the device is still in use, so if your goal was to disconnect or format it, you still cannot. umount -f is meant for NFS with the server down, where processes are blocked in state D; on a local system it can leave writes unflushed. And fuser -k sends SIGKILL, with no chance to save anything: check first with fuser -vm whom you are about to kill.

A fourth case, little known but frequent: if the file system looks free and is nevertheless busy, check whether you have another mount on top (a bind mount, a container, an overlay) with findmnt -R /var/lib/meteora, which shows the mount subtree recursively.

A bind mount makes an existing directory visible at a second point of the tree. It copies nothing and creates no link: it is the same file system mounted twice.

Now both paths lead to the same inodes. The difference from a symbolic link is substantial:

That last row is the most useful in practice: you can expose a data directory to a service read-only, without touching the original, and along the way solve the puzzle of section 7 — seeing what was hidden under a mount — without unmounting anything:

The service that serves /srv/publishing/data cannot write; the ingestor, which uses the original path, can. It is the mechanism behind systemd's ReadOnlyPaths= (module 7) and Docker's read-only volumes.

Bind mounts are also the doorway to a bigger concept: mount namespaces. Linux allows each process to have its own mount table, so that /var/lib/meteora can mean different things to two processes on the same machine. It is the foundation of containers, which combine it with pivot_root, layered file systems (overlayfs) and cgroups; you get the full picture in Containers: Namespaces and cgroups. Here it is enough to take away that the mount table is not necessarily global, and that /proc/<pid>/mountinfo tells you which one each process sees.

We have reached the piece that explains everything. Consider this program:

Four incompatible file systems: one with inodes and extents on an SSD, another living in the page cache, another that generates the content at the moment of reading by running kernel code, and another that translates every operation into network packets. And yet, the same four lines of code work on all four.

What makes that possible is the VFS (Virtual File System), an abstraction layer inside the kernel:

The idea is exactly that of polymorphism in object-oriented programming, implemented in C with tables of function pointers. The VFS defines which operations exist; each file system supplies its implementation. When a read() arrives, the VFS does not know how to read anything: it looks at the file's operations table and calls the corresponding pointer, which points to ext4_file_read_iter, to shmem_file_read_iter or to the procfs module's function.

The VFS's four objects, worth having clear because they explain concrete behaviors:

The relationships among them answer questions you may have been asking yourself. Several dentries can point to the same inode: those are hard links, two names and one file. Several file objects can point to the same inode: two processes that open the same file have different file objects — each with its own offset and its own mode — over a single inode, and out of that comes all the behavior of descriptors, fork and dup in 04-04. And the VFS's inode is not the disk inode, but its in-memory representation common to all file systems: a procfs file has a VFS inode even though nothing exists on disk.

An example of the operations table, simplified from the real kernel:

read() in user mode ends up, after the switch to kernel mode of 01-06, in vfs_read(), which essentially does file->f_op->read_iter(...). One indirection through a function pointer, and that is where the path forks towards ext4, tmpfs or NFS.

What the VFS provides, in summary: a single implementation of the system calls for all file systems; shared caches (pages, dentries, inodes) that benefit them all; the ability to add a new file system as a module without touching the rest of the kernel; and the transparency that lets a path cross from ext4 to tmpfs halfway along without any program noticing. It is the same extended-machine philosophy of 01-01, applied one layer further down.

With the VFS understood, the "odd" file systems stop being odd. They all implement the same interface; what changes is where they get the data from.

procfs is the most peculiar: its files do not exist. When you run cat /proc/2841/status, the read() arrives via the VFS at a kernel function that walks process 2841's task_struct and formats text on the fly. That is why ls -l /proc/2841/status shows size 0 — the kernel does not know how many bytes it will produce until it produces them — and why the content changes between two consecutive reads. Everything we have used from modules 2 and 3 — /proc/<pid>/maps, /proc/<pid>/stack, /proc/<pid>/fd, /proc/mounts — is this: a kernel query interface disguised as files, so it can be used with cat, grep and awk instead of with a special API. It is one of Linux's best ideas.

tmpfs is a complete file system whose store is the page cache. It is blazingly fast, because a write is a copy in RAM; it is volatile, and that is a feature rather than a defect — it is why /run/meteora/readings.fifo and /run/meteora/api.sock are there (04-02); it grows and shrinks dynamically up to the size= limit; and it can go to swap under memory pressure, unlike a classic RAM disk which pins it down.

And here a circle from module 3 closes. Meteora's cache, /dev/shm/meteora-cache, we created with shm_open() + mmap() as POSIX shared memory. Now you can see what it really is: shm_open() is an open() on a tmpfs file mounted at /dev/shm, and mmap() maps its pages into several processes. POSIX shared memory on Linux is implemented on top of the file system, and that is why you can run ls -l, chmod and rm on it like on any other file:

128 MiB of cache that are, at the same time, a file and a shared memory segment. There is no contradiction: it is the VFS doing its job.

The VFS allows something more ambitious: the file system being on another machine. The client implements the VFS operations by translating them into network requests.

Mounting them is like mounting anything else, which is precisely the point of the VFS:

The three problems to know about before using them in production:

1. Latency changes everything. A local stat() costs microseconds; over NFS, a network round trip: between 0.1 and 5 ms. An ls -l of a directory with 1,000 files does 1,000 stat() calls, which locally is milliseconds and over NFS can be five seconds. The cause is never the bandwidth, but the number of round trips.

2. The semantics are not the same. POSIX guarantees that a completed write() is immediately visible to any other process; NFS uses weak coherence, caches attributes and data, and another client may take seconds to see the change. Worse still, file locking (flock, which we will see in 04-04) is notoriously fragile over NFS. Practical rule: do not put anything on NFS that depends on locks or on writes coordinated between machines, databases in particular.

3. hard versus soft. With hard (the default), if the server stops responding the processes are left blocked indefinitely in state D, not even responding to SIGKILL: the clinical picture of module 3 and the reason for the hung task detector of 03-06. The alternative soft returns an error after a timeout, but it can corrupt data if the error arrives in the middle of a write. The balanced choice is hard plus intr (or NFSv4, which allows interrupting), and always mounting with nofail.

Meteora uses NFS only for the read-only historical archive at /mnt/historical: data that no longer changes, with no locks and no concurrent writes. /var/lib/meteora, where the ingestor writes constantly, is on local storage over RAID 1, and you now have the three technical reasons why that decision is the right one.

Using /dev/sdX in /etc/fstab. Kernel names change with the detection order. Always use UUID=, and remember that the UUID changes when you reformat.

Editing fstab without validating. A badly written line leaves the machine in emergency mode at the next boot, and if it is a remote server, with no access. Backup, findmnt --verify and mount -a before rebooting. Always.

Writing into the mount point before mounting. The files end up hidden under the mount, taking up invisible space on the partition underneath. Check with a bind mount of /.

Formatting the wrong device. mkfs does not ask. The three verification commands (lsblk, blkid, findmnt --source) plus the dry run with -n cost ten seconds and prevent the most expensive accident in the profession.

Shrinking a logical volume in the wrong order. When extending: lvextend and then resize2fs. When shrinking: resize2fs first and then lvreduce. Reversing it when shrinking cuts through data. Use lvresize -r, which gets it right on its own.

Leaving LVM snapshots alive indefinitely. They cost 20-40 % of write performance and, if they fill up, they are invalidated and you lose the backup. Create, use and remove.

Reaching for umount -l as a reflex answer to "target is busy". It hides the problem: the file system is still in use even though it has disappeared from the tree. Diagnose first with lsof or fuser, and stop the service.

Mounting network file systems without nofail. A powered-off NAS blocks the server's boot. And with hard, a downed server leaves processes in D that do not die even with kill -9.

Tip: mount data volumes with nosuid,nodev,noexec — three words that close three attack vectors — and use findmnt instead of plain mount, which shows the tree, filters, verifies and returns exit codes that are useful in scripts.

On any machine, run lsblk -f, findmnt, blkid and cat /proc/mounts, and answer: (a) how many file systems are mounted and how many have a real device behind them? (b) what options does /tmp have and what does each one protect against? (c) how much RAM are the tmpfs instances consuming right now? (d) create a file in /dev/shm and locate where its memory shows up in free -h. Explain the result of (d) with what you know about the VFS and the page cache.

You are going to install a meteo-02 with a 1 TB NVMe and two 4 TB SATA disks for the historical archive. Design the full scheme: partition table (justifying MBR or GPT), partitions with their sizes and file systems, whether or not to use LVM, and the complete /etc/fstab with the six fields and the mount options justified one by one. Explain what each separation protects against and what would happen if you did not make it. The server will run Meteora's three services and store five years of history.

/var/lib/meteora is on a 600 GB logical volume and the ingestor writes non-stop. Write the complete nightly backup script that produces a consistent copy without stopping the service: justified sizing of the snapshot, the command sequence with its error handling, the check that the snapshot has not overflowed, and guaranteed cleanup even if the script fails halfway. Explain as well why sync before creating the snapshot is not quite enough and what would be needed for a perfect backup.

Solution 1

(a) wc -l < /proc/mounts gives the total and grep -c '^/dev/' /proc/mounts those with a real device. On a typical system you get between 25 and 40 mounts, of which only 3 to 6 have a real device; everything else is procfs, sysfs, tmpfs, devtmpfs, cgroupfs, devpts, securityfs... The file system you see is mostly an in-memory construction, and that is the best practical demonstration of what the VFS is for.

(b) findmnt /tmp gives something like rw,nosuid,nodev,noexec,relatime,size=2097152k. nosuid prevents escalation with a setuid binary dropped there — /tmp is writable by everybody, so it is the natural place to try it; nodev prevents creating a home-made /dev/sda and reading the raw disk; noexec prevents directly executing a downloaded binary (although it does not block bash script.sh); and size=2G limits how much RAM it can consume.

(c) df -h -t tmpfs: the "Used" column of each tmpfs is RAM occupied right now, and it usually comes to a few hundred MiB across /run, /dev/shm and /tmp.

(d)

Explanation. tmpfs has no device: its pages are page cache pages (02-04), so free counts them under buff/cache. The difference from an ordinary cache is crucial: tmpfs pages cannot be discarded, because there is no copy on any disk to read them back from; they can only go to swap. That is why a tmpfs with no size= can end up exhausting the system's memory and triggering the OOM killer of 02-04, and why every tmpfs on the system carries a limit.

Solution 2

Partition table: GPT, because the 4 TB disks exceed MBR's 2 TiB limit; it also brings redundancy and CRC.

Leaving unallocated space in the VG is a design decision, not an oversight: since LVM extends live, it is better to distribute when you know where it is needed than to guess on installation day.

Justification of the non-obvious options: errors=remount-ro on / makes an I/O error switch the volume to read-only instead of letting it go on corrupting itself; nosuid,nodev on /var because there is no legitimate reason for a setuid binary there, but without noexec because some package managers run things from /var/lib; noexec yes on /var/lib/meteora and /srv/historical, which are pure data; pass=1 only on /, 2 on the rest and 0 on swap and tmpfs, which have no fsck; nofail on /srv/historical so that a RAID that fails to assemble does not stop the boot; and /tmp as a 4 GB tmpfs, fast, self-cleaning and capped so it does not exhaust the RAM.

What would happen without the separations. Without a separate /var, a runaway log would fill / and you could not log in to fix it. Without a separate /var/lib/meteora, you could not apply noatime or a tuned mkfs to it. Without a separate /boot, encrypting the root disk would be far more complicated.

Solution 3

Sizing. The snapshot stores the original blocks of everything that changes while it lives. Meteora writes about 17.3 MB a day, that is 720 KB/hour; if the backup takes 30 minutes, about 360 KB will change. With 10 GB there is a safety factor of 28,000× against an anomalous spike, and it is still 1.6 % of the volume. Oversizing here is cheap; falling short means losing the backup.

The script's three important decisions: set -euo pipefail aborts at the first error instead of carrying on with an incomplete backup; trap cleanup EXIT guarantees that the snapshot is removed even if the script fails or is killed — without this, a forgotten snapshot penalizes writes for days and eventually overflows; and the subsequent data_percent check warns you if the snapshot filled up during the copy, which is what turns the script into a backup rather than an illusion.

Why sync is not quite enough. sync flushes the kernel's page cache to the device, so the snapshot picks up everything the kernel had pending. But it does not empty user-space buffers: if the ingestor uses the C library's FILE*, it may have data in its own buffer that has not even reached the kernel yet (04-04). Hence the preceding systemctl reload, which asks it to close and reopen its files with fsync(). A perfect backup also requires the application to have a consistent point — a closed transaction, a complete day's file — and not just flushed bytes; with a format of 24-byte records appended at the end, any interruption leaves at most one incomplete reading, which the reader detects from the size. It is the difference between byte consistency and application consistency, and we will come back to it in 04-05.

A partition is a contiguous range of LBAs declared as an independent unit, and the table describing them is either MBR or GPT. The choice is no longer a matter of opinion: MBR tops out at 2 TiB and 4 primary partitions and has no redundancy; GPT reaches 8 ZiB, allows 128 partitions, keeps a copy of the table at the end of the disk verified with CRC32 and protects against accidental formatting with a protective MBR. You partition to isolate the filling, apply different policies, use different file systems and satisfy the boot requirements, and separating /var has a devastating argument behind it: a full / is a server you cannot get into to fix it. LVM adds the layer partitions lack: PVs, VGs and LVs allow extending live (lvextend + resize2fs, in that order; the reverse for shrinking), spreading a volume across disks and moving data without stopping anything; and its snapshots solve the consistent backup of /var/lib/meteora without stopping the ingestor, at the price of a 20-40 % write penalty and the risk of being invalidated if they fill up.

Mounting is the module's central operation: resolve the mount point, read the superblock, check the state, create the superblock and the vfsmount in memory and hook the dentry so that path resolution jumps into the new tree. Two facts follow from that last step: that mounting over a non-empty directory hides its content without deleting it, and that the mount table in /proc/mounts is the system's truth. Stable identification is with UUID=, never with /dev/sdX, and /etc/fstab has six fields where pass is 1 on the root, 2 on the rest and 0 on whatever has no fsck — with the mandatory procedure of backup, findmnt --verify and mount -a before rebooting.

The mount options are free hardening: nosuid cuts off escalation via a setuid binary, nodev prevents reading the raw disk while bypassing permissions, noexec adds one more layer, and errors=remount-ro stops the damage from a dying disk. That is why /var/lib/meteora goes with noatime,nosuid,nodev,noexec. The "target is busy" has four causes — open file, cwd, mmap and a mount on top — is diagnosed with lsof or fuser -vm, and is resolved by stopping the service: umount -l unhooks but does not release, and fuser -k kills without warning. Bind mounts expose the same tree at two points with different options, allow you to see what is hidden under a mount and are the antechamber to the mount namespaces of 06-02.

And the underlying explanation for all of it is the VFS: four objects — superblock per mount, inode per file, dentry per name, file per open — and tables of function pointers that make vfs_read() end up in ext4_file_read_iter, in shmem_file_read_iter or in the NFS client, as the case may be. Thanks to it, /proc can generate its files as you read them — hence the size 0 and the changing content — tmpfs can be a file system whose store is the page cache — and that is why /dev/shm/meteora-cache is at once a file and POSIX shared memory — and NFS can put a file system at the other end of the network, with its three drawbacks: latency per round trip, weak coherence that breaks locks, and hard leaving processes in D when the server goes down.

We now have the complete map of storage: we know what a file is, how it gets a name, and how the tree we reach it through is assembled. What we have not done yet is use it from a program. What exactly happens when open() returns the number 3? Why do parent and child share the offset after a fork but two open calls on the same file do not? How does the shell implement the > redirection and that 2>&1 everybody copies without understanding? And how does the aggregator make sure the hourly averages it publishes are never read half-written?

That is what we will see in File Management.