We closed module 3 with an uncomfortable admission: for three modules we had been writing /var/lib/meteora/readings/2026-08-31.dat as if that string of text were something self-evident, when in module 2 we left storage in a very different place. There, an NVMe SSD was a vector of blocks numbered 0 to N, addressable by LBA, with no names, no folders, no sizes and no owners. A giant array of 512 or 4,096 bytes per slot. Nothing more.
Between that vector of blocks and the path /var/lib/meteora/readings/2026-08-31.dat sits one of the most successful abstractions in computing: the file system. It is the piece that turns "block 8,394,271" into "yesterday's readings file", that remembers who created it and when, that knows it takes up 17 MB spread over thousands of scattered blocks, and that lets a C program read it without knowing anything at all about LBAs, sectors or geometry.
This lesson builds that abstraction from the ground up. We are going to define precisely what a file is, see what metadata it carries and where that metadata lives, dissect the inode — the central structure of every UNIX system — walk through the physical layout of a file system on the device, calculate why the block size is a trade-off and not an accident, and finish by comparing the families of file systems that exist so we can decide, with arguments, which one deserves /var/lib/meteora.
Contents
- From the raw block to the file: what problem the abstraction solves
- What a file is: content, metadata and name
- The seven UNIX file types
ls -landstatinterpreted field by field- The inode: what it contains and what it does not
- Inode numbers,
ls -iand exhaustion withdf -i - Physical layout of a file system on the device
- Block size and its trade-off, calculated
- Timestamps:
atime,mtime,ctime,crtimeand the reason fornoatime - Families of file systems compared
- Meteora's decision for
/var/lib/meteora
From the raw block to the file: what problem the abstraction solves
Imagine for a moment that Meteora had no file system and worked directly on /dev/nvme0n1, the block device from module 2. The ingestor would have to solve, all by itself, this list of problems:
| Problem | What the ingestor would have to do without a file system |
|---|---|
| Location | Remember which LBA today's data starts at, and store that somewhere... where? |
| Growth | Know how many blocks to reserve in advance, because it cannot "grow" over a neighbour |
| Free space | Keep its own accounting of which blocks are used and which are not |
| Names | Invent its own index translating "31 August" into a block number |
| Concurrency | Coordinate with the aggregator and with meteo-api so they do not step on each other's blocks |
| Permissions | There are none: anyone with access to the device reads and writes everything |
| Persistence of the accounting | If the power fails while it updates its index, it loses everything |
Every program on the server would have to solve all seven, each one its own way, and none could cooperate with the others. It is exactly the situation we described in 01-01 when we talked about the operating system as an extended machine: without it, every application reimplements the hardware.
The file system solves all seven at once with a single idea:
A file is a named sequence of bytes, with a size and an owner, that the system stores wherever it likes and the program reads as if it were continuous.
The four key phrases in that definition deserve attention:
- Sequence of bytes. In UNIX a file has no internal structure known to the system. There are no "records", no "fields", no types. That
2026-08-31.datcontains 720,000 24-byteReadingstructs is an agreement among Meteora's own programs, invisible to the kernel. Other historical systems (IBM's, VMS) did impose a record structure, and the industry eventually decided UNIX's flat model was right, because of its simplicity. - Named. The name is the handle by which the user grabs the file. In section 5 we will see the surprise: the name is not inside the file.
- Wherever it likes. The system decides which blocks to use. That is what we will study in Space Allocation, Journaling and Integrity.
- As if it were continuous. This is the magic.
2026-08-31.datoccupies 4,219 blocks that may be scattered all over the SSD, and yetread()delivers them in order as a stream of bytes. It is the same class of illusion as the virtual memory of module 2: contiguous logical addresses over scattered physical storage.
In fact the parallel with virtual memory is so exact that it is worth pinning down, because it will save you effort throughout the module:
| Virtual memory (02-04) | File system (module 4) |
|---|---|
| The process's virtual address space | The file as a sequence of bytes 0..N |
| Page (4 KiB) | Logical block (4 KiB) |
| Page frame in RAM | Physical block on the device |
| Page table | Inode pointers/extents |
| The MMU translates virtual → physical | The file system translates offset → LBA |
| A page fault brings the page in from disk | read() brings the block into the page cache |
It is the same idea applied twice: a translation table that turns an orderly logical space into a disorderly physical one.
What a file is: content, metadata and name
A file has three parts, and they live in three different places. This separation is the key to everything else:
graph LR
subgraph DIR["Directory (04-02)"]
N["name: 2026-08-31.dat<br/>inode: 1180934"]
end
subgraph INO["Inode no. 1180934"]
M["type, permissions, owner,<br/>size, dates, pointers"]
end
subgraph DAT["Data area"]
D["4,219 blocks<br/>holding the 17,280,000 bytes"]
end
N -->|points to| M
M -->|points to| D
- The content: the bytes themselves, in the device's data area.
- The metadata: everything the system knows about the file. It lives in the inode.
- The name: it lives in the directory that contains it, not in the file. We will develop this in Directory Structures, but you need to know it already to understand the inode.
A file's typical metadata, with what each item means:
| Metadata | What it is | Example in Meteora |
|---|---|---|
| Type | Regular, directory, symbolic link... | Regular |
| Size | Bytes of content | 17,280,000 |
| Blocks | 512 B blocks actually occupied | 33,760 |
| Owner (UID) | Owning user | meteora (UID 990) |
| Group (GID) | Owning group | meteora (GID 990) |
| Permissions | 12 mode bits | 0640 |
| Link count | How many names point to this inode | 1 |
| Timestamps | Access, modification, change, creation | see section 9 |
| Data pointers | Where the blocks are | extents (04-05) |
Notice something you can already sense: the name does not appear in that list. It is not an oversight; it is UNIX's central design decision, and out of it come hard links, deferred deletion and half a dozen behaviors that are surprising until you understand this.
The seven UNIX file types
In UNIX, the phrase "everything is a file" is meant seriously. The same interface — open, read, write, close — works for a data file, a keyboard, a pipe or a network connection. What changes is the type, a 4-bit field in the inode:
Symbol in ls -l |
Type | What it is | Meteora example |
|---|---|---|---|
- |
Regular | Sequence of bytes on disk | /var/lib/meteora/readings/2026-08-31.dat |
d |
Directory | Table of (name, inode) pairs | /var/lib/meteora/readings/ |
l |
Symbolic link | A file containing a path | /var/lib/meteora/readings/today.dat |
b |
Block device | Block access, cached | /dev/nvme0n1, /dev/md0 |
c |
Character device | Byte access, uncached | /dev/null, /dev/random, /dev/tty |
p |
FIFO (named pipe) | A channel in the file system | /run/meteora/readings.fifo |
s |
Socket | Local communication endpoint | /run/meteora/api.sock |
The last three are old friends from module 3: we used the FIFO at /run/meteora/readings.fifo in Inter-Process Communication (IPC), and UNIX domain sockets too. The difference is that there we saw them as IPC mechanisms and here we see them as file system entries: they have an inode, permissions and a name, but their content is not on disk — a FIFO has a buffer in kernel memory, a socket has a network queue. They take up one inode and zero data blocks.
The same goes for block and character devices, which we saw in Device Management: their inode holds no data pointers, it holds the (major number, minor number) pair that identifies the driver. A device file is, literally, an inode with a couple of integers inside.
ls -l and stat interpreted field by field
Let us really look at it. This is the listing of Meteora's directory on meteo-01:
$ ls -l /var/lib/meteora/readings/ /dev/nvme0n1 /dev/null /run/meteora/ -rw-r----- 1 meteora meteora 17280000 Aug 31 23:59 2026-08-31.dat -rw-r----- 1 meteora meteora 17280000 Sep 1 12:40 2026-09-01.dat lrwxrwxrwx 1 meteora meteora 14 Sep 1 00:00 today.dat -> 2026-09-01.dat drwxr-x--- 2 meteora meteora 4096 Sep 1 00:00 archive brw-rw---- 1 root disk 259, 0 Sep 1 08:12 /dev/nvme0n1 crw-rw-rw- 1 root root 1, 3 Sep 1 08:12 /dev/null prw-r----- 1 meteora meteora 0 Sep 1 08:13 /run/meteora/readings.fifo srwxr-xr-x 1 meteora meteora 0 Sep 1 08:13 /run/meteora/api.sock
The fields, from left to right:
- First character: the type from the previous table.
-,l,d,b,c,p,s. At a glance you already know what each thing is. - The next nine characters: the permissions, which we will look at in File Security and Permissions.
- The number after the permissions: the link count. It is 1 for ordinary files and 2 for the
archivedirectory (04-02 explains why). - Owning user and group:
meteora meteora. - Size: 17,280,000 bytes for the readings. But look at
/dev/nvme0n1: where the size ought to be it says259, 0. Those are the major and the minor. A device file has no size because it has no content; in its placelsshows the pair that identifies the driver. And the FIFO and the socket show0, for the same reason. - Date: by default the
mtime, not the creation time nor the last access time (section 9). - Name, with the
-> targeton the symbolic link.
ls -l is a summary. To see everything you use stat:
$ stat /var/lib/meteora/readings/2026-08-31.dat File: /var/lib/meteora/readings/2026-08-31.dat Size: 17280000 Blocks: 33760 IO Block: 4096 regular file Device: 9,0 Inode: 1180934 Links: 1 Access: (0640/-rw-r-----) Uid: ( 990/meteora) Gid: ( 990/meteora) Access: 2026-09-01 06:00:11.482913711 +0200 Modify: 2026-08-31 23:59:58.117204339 +0200 Change: 2026-08-31 23:59:58.117204339 +0200 Birth: 2026-08-31 00:00:00.004118220 +0200
Field by field, with what you need to understand about each one:
- Size: 17280000. The file's logical bytes: exactly 720,000 readings × 24 bytes. It is the number
lseek(fd, 0, SEEK_END)returns. - Blocks: 33760. Here is a classic trap:
statcounts 512-byte blocks, always, regardless of the file system's real block size. 33,760 × 512 = 17,285,120 bytes occupied on disk, somewhat more than the 17,280,000 logical ones. The difference is 5,120 bytes: 4 KiB for the partially used last block plus the metadata of the extent tree. When "blocks × 512" is smaller than the size, that is the signature of a sparse file (we will see it in 04-04). - IO Block: 4096. The preferred block size for reads; reading in multiples of this number saves the kernel extra work.
- Device: 9,0. Major 9, minor 0:
/dev/md0, the RAID 1 we assembled in 02-05. This pair identifies which file system the inode lives in, and the (device, inode) pair is the only thing that uniquely identifies a file across the whole machine. - Inode: 1180934. The inode number. Section 6 is devoted to it.
- Links: 1. A single name points to this inode (04-02).
- Access (0640). Twelve mode bits, shown in octal and symbolically (04-06).
- Uid/Gid 990. The
meteorauser and group. Careful: the inode stores numbers, not names; the translation intometeorais done bystatconsulting/etc/passwd. - The four dates: section 9.
The inode: what it contains and what it does not
The inode (index node) is the data structure that is the file. Everything else is a reference to it. It lives on disk, in a reserved area, and has a fixed size: 256 bytes on a modern ext4 (128 on old ones, configurable at format time).
Its contents, grouped:
| Group | Fields | Approx. bytes |
|---|---|---|
| Identity | Type (4 bits) + permissions (12 bits), UID, GID | 8 |
| Size | Logical size in bytes (64 bits), blocks occupied | 12 |
| Links | Count of names referencing it | 2 |
| Times | atime, mtime, ctime, dtime, crtime (with nanoseconds) | 40 |
| Data | 60 bytes of pointers/extents (04-05) | 60 |
| Extras | Flags (chattr), version, checksum, extended attributes |
the rest |
And now the important part, which is what is not there:
The file's name is not in the inode. Neither is the path, nor the directory it belongs to, nor anything relating it to
/var/lib/meteora/readings/2026-08-31.dat.
Inode 1180934 does not know what it is called. It knows it is a regular file of 17,280,000 bytes belonging to UID 990, that there is 1 name somewhere pointing at it, and which blocks hold its data. Nothing else.
This is not a limitation: it is the design decision half this module hangs from. Its consequences, which we will unfold as we go:
| Consequence | Where it is explained |
|---|---|
| One and the same file can have several names (hard links) | 04-02 |
| Renaming is instantaneous: only a directory entry changes | 04-02 |
Deleting is unlink: removing one name, not destroying the file |
04-02 |
| An open file with no name at all still exists | 04-02, 04-04 |
| The name has no permissions; the permissions are in the inode | 04-06 |
A file in use can be replaced atomically with rename() |
04-04 |
Keep this sentence: the inode is the file; the name is just a label stuck on from outside.
Inode numbers, ls -i and exhaustion with df -i
Every inode has a unique number within its file system. You can see it with ls -i:
Two files in the same file system with the same inode number are the same file. Two files in different file systems may share a number without having anything to do with each other: that is why the real identity is the (device, inode) pair, st_dev + st_ino in stat. It is exactly what find -samefile checks and what rsync uses so it does not copy the same linked content twice.
Now the practical part. In ext4, the number of inodes is fixed at format time and cannot be increased afterwards. They are reserved in advance, with a default ratio of one inode per 16 KiB of capacity:
$ df -h /var/lib/meteora Filesystem Size Used Avail Use% Mounted on /dev/md0 196G 41G 146G 22% /var/lib/meteora $ df -i /var/lib/meteora Filesystem Inodes IUsed IFree IUse% Mounted on /dev/md0 13107200 1834 13105366 1% /var/lib/meteora
Interpretation: the volume has 13,107,200 inodes (200 GiB ÷ 16 KiB) and uses only 1,834, because Meteora stores one large file per day and has been running for about five years. With that policy, in 100 years it would use 36,500 inodes: 0.28 %.
The other way round, however, produces the most bewildering failure in systems administration:
$ df -h /var/spool/cache Filesystem Size Used Avail Use% Mounted on /dev/sdb1 50G 12G 36G 26% /var/spool/cache ← 74 % free! $ df -i /var/spool/cache Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sdb1 3276800 3276800 0 100% /var/spool/cache ← 0 free $ touch /var/spool/cache/test touch: cannot touch 'test': No space left on device
"No space left on device" with 36 GB free. The error is ENOSPC and it is literal from the kernel's point of view: there is no inode space left. A process that creates millions of tiny files — caches, sessions, mail queues — exhausts the inode reserve long before the blocks. Diagnostic rule: faced with an ENOSPC, always run both df -h and df -i; if the first explains nothing, the second will.
To find the culprit:
$ sudo find /var/spool/cache -xdev -type f -printf '%h\n' | sort | uniq -c | sort -rn | head -5 2984112 /var/spool/cache/sessions 12043 /var/spool/cache/tmp
The command counts files per directory: -xdev avoids crossing into other file systems, -printf '%h\n' prints each file's directory, and sort | uniq -c | sort -rn groups and sorts. Almost three million files in sessions: there is the problem.
The fix is not to enlarge the disk, because inodes do not grow: you have to delete files or reformat with mkfs.ext4 -i 4096 (one inode per 4 KiB, four times as many) or -N 8000000 (an explicit number). This is one of the arguments in favor of XFS and Btrfs, which allocate inodes dynamically and do not suffer from this problem.
Physical layout of a file system on the device
We now know what an inode is. Where is it physically? Let us open the device up. The hierarchy of units, from smallest to largest:
| Unit | Typical size | Who defines it |
|---|---|---|
| Sector | 512 B (logical) / 4,096 B (physical) | The disk hardware |
| Logical block | 1, 2 or 4 KiB | The file system, at format time |
| Block group | 128 MiB in ext4 | The file system |
| File system | The whole partition | The administrator |
The sector is the smallest unit the device knows how to read or write; we saw it in 02-05. The logical block is the smallest unit the file system knows how to allocate: even though the disk can read 512 bytes, ext4 never allocates less than one block to a file.
An ext4 is divided into block groups of 128 MiB, each with its own accounting. The reason is performance and comes straight out of module 2: keeping together the metadata and the data that are used together reduces head movement on an HDD and improves locality on an SSD.
graph TB
subgraph FS["/dev/md0 — 200 GiB ext4, 1,600 groups of 128 MiB"]
BOOT["Block 0<br/>1 KiB of boot area"]
subgraph G0["Block group 0"]
SB["Superblock<br/>(1 block)"]
GD["Group descriptors<br/>(N blocks)"]
BB["BLOCK<br/>bitmap<br/>(1 block)"]
IB["INODE<br/>bitmap<br/>(1 block)"]
IT["Inode table<br/>(512 blocks)"]
DZ["DATA AREA<br/>(~31,000 blocks)"]
end
G1["Group 1<br/>(same structure)"]
GN["... Group 1,599"]
end
BOOT --> SB --> GD --> BB --> IB --> IT --> DZ --> G1 --> GN
Piece by piece:
The superblock. It is the file system's identity card, and it takes up a single block. It contains the total number of inodes and blocks, how many are free, the block size, the inode size, the number of blocks per group, the UUID, the label, the date of the last mount, the mount counter and the state (clean or dirty, which will be decisive in 04-05). Without a superblock nothing can be mounted, because you do not even know where the inodes begin. That is why ext4 keeps backup copies in several groups:
$ sudo dumpe2fs /dev/md0 | grep -i superblock Primary superblock at 0, Group descriptors at 1-13 Backup superblock at 32768, Group descriptors at 32769-32781 Backup superblock at 98304, Group descriptors at 98305-98317 Backup superblock at 163840, ...
If the primary one is corrupted, you recover with sudo e2fsck -b 32768 /dev/md0, which tells e2fsck to use the backup at block 32768. It is a command worth writing down: it saves volumes that looked lost.
The group descriptors. An array with one entry per group, saying where that group's bitmaps and inode table are and how many free elements it has. It is the index that lets you find everything else.
The block bitmap. One bit per block in the group: 1 = used, 0 = free. With 4 KiB blocks, a 128 MiB group has 32,768 blocks, which is 32,768 bits = 4,096 bytes: exactly one block. That is no coincidence; the group size is chosen precisely so that its bitmap fits in one block.
The inode bitmap. The same thing for the group's inodes.
The inode table. The array of inodes proper. If a group has 8,192 inodes of 256 bytes, it takes up 2 MiB = 512 blocks. This is where an inode's position is computed from its number, with pure arithmetic and without searching any index:
/* Locate inode 1180934 in an ext4 with 8,192 inodes per group */
unsigned group = (1180934 - 1) / 8192; /* = 144 → group 144 */
unsigned index = (1180934 - 1) % 8192; /* = 1157 → position 1157 */
off_t position = inode_table_start[144] + (off_t)1157 * 256;Two divisions and one multiplication: that is all it costs to go from an inode number to its position on disk. Inodes are numbered from 1, hence the - 1. This is the deep reason why an inode is a number and not a name: looking up a name would require walking a table; a number is resolved with arithmetic in nanoseconds.
The data area. Everything else: the blocks holding the files' bytes.
You can see it on your own machine with dumpe2fs:
$ sudo dumpe2fs -h /dev/md0 | head -20 Filesystem volume name: meteora-data Filesystem UUID: 9f3a1c22-7d4e-4a51-b7c8-2e5f0a1d6b93 Filesystem features: has_journal ext_attr dir_index extent 64bit Filesystem state: clean Inode count: 13107200 Block count: 52428800 Free blocks: 41156923 First block: 0 Block size: 4096 Blocks per group: 32768 Inodes per group: 8192 Inode size: 256
Every line is a real superblock field. Block count: 52428800 × 4,096 = 200 GiB. Filesystem state: clean means it was unmounted properly. And dir_index and extent are features that will show up in 04-02 and 04-05.
Block size and its trade-off, calculated
The block size is chosen at format time and cannot be changed afterwards. It is a genuine trade-off, and it shows up in the numbers.
The cost of a large block: internal fragmentation. Since allocation happens in whole blocks, each file's last block is left partially empty, and that space is lost. On average half a block per file is wasted.
The cost of a small block: more blocks to manage. More entries in the metadata, more allocation work, and smaller I/O requests.
With Meteora's real files (17,280,000 bytes):
| Block size | Blocks needed | Space occupied | Waste | Bitmap for 200 GiB |
|---|---|---|---|---|
| 1 KiB | 16,875 | 17,280,000 B | 0 B (exact) | 25 MiB |
| 2 KiB | 8,438 | 17,281,024 B | 1,024 B | 12.5 MiB |
| 4 KiB | 4,219 | 17,281,024 B | 1,024 B | 6.25 MiB |
| 16 KiB | 1,055 | 17,285,120 B | 5,120 B | 1.6 MiB |
| 64 KiB | 264 | 17,301,504 B | 21,504 B | 0.4 MiB |
For 17 MB files the waste is irrelevant in every case: 21 KB out of 17 MB is 0.12 %. But the number of blocks to manage changes by a factor of 64.
Now the opposite scenario, a directory with a million 800-byte files (sessions, not readings):
| Block size | Space occupied by 1,000,000 files | Waste |
|---|---|---|
| 1 KiB | 1,024 MB | 224 MB (22 %) |
| 4 KiB | 4,096 MB | 3,296 MB (80 %) |
| 64 KiB | 65,536 MB | 64,736 MB (99 %) |
With 64 KiB blocks, a million 800-byte files would occupy 64 GB to store 800 MB. The very same block size that was irrelevant for the readings is catastrophic here.
Why 4 KiB is the universal standard, and not an arbitrary convention:
- It matches the page size (module 2). The kernel's page cache works in 4 KiB pages; if the block is the same size, one block = one page and nothing has to be split or joined. A block larger than the page cannot be mapped directly with
mmap(), which is precisely what theaggregatordoes with2026-08-31.dat(02-04). In fact, Linux does not support blocks larger than the page size, so on x86-64 the maximum is 4 KiB. - It matches the physical sector of modern disks (Advanced Format, 4Kn), so no write triggers the read-modify-write cycle.
- It balances the waste for the real distribution of file sizes on a typical system.
Practical rule: use 4 KiB unless you have a measured reason not to. The reason to go down to 1 KiB is a volume dedicated to millions of tiny files; to go up you would have to move to XFS on architectures with larger pages, or to file systems with bigalloc.
Timestamps: atime, mtime, ctime, crtime and the reason for noatime
Four dates, and three of them get confused constantly:
| Timestamp | Name | Updated when... | Seen with |
|---|---|---|---|
| atime | access time | The content is read | ls -lu, stat |
| mtime | modify time | The content changes | ls -l (by default) |
| ctime | change time | The inode changes | ls -lc, stat |
| crtime | creation time | The file was created | stat (ext4, XFS, Btrfs) |
The most frequent mistake is believing that ctime is creation time. It is not: it is change time, the moment any piece of metadata changed. Changing the permissions with chmod, the owner with chown or creating a hard link modifies the ctime but not the mtime, because the content has not changed. And since it cannot be set backwards (not even touch -d touches it), the ctime is the timestamp a forensic analyst looks at: whoever tampers with a file can fake atime and mtime, but in doing so they update the ctime, giving themselves away. We will come back to it in module 5.
A table of what each operation touches on 2026-08-31.dat:
| Operation | atime | mtime | ctime |
|---|---|---|---|
cat 2026-08-31.dat |
Yes | No | No |
echo data >> 2026-08-31.dat |
No | Yes | Yes |
chmod 640 2026-08-31.dat |
No | No | Yes |
chown meteora: 2026-08-31.dat |
No | No | Yes |
mv 2026-08-31.dat old.dat |
No | No | No¹ |
ln 2026-08-31.dat copy.dat |
No | No | Yes² |
¹ Renaming changes the directory, not the file's inode. ² It changes the link count, which is in the inode.
Why noatime improves performance
Here is the practical problem. Updating the atime means writing to disk when you read a file. It sounds absurd, and it is: it turns a read-only operation into a read-and-write one.
The numbers for Meteora. meteo-api serves about 2,000 queries an hour, each one reading the day's file. With strict atime, that is 2,000 writes an hour to the inode of a file nobody has modified: 48,000 daily metadata writes, plus their corresponding journal entry (04-05), on a RAID 1 where every write goes to both disks. And on an SSD, write cycles spent for nothing.
The available mount options, which we will apply in Partitions, Mounting and the Virtual File System:
| Option | Behavior | Cost |
|---|---|---|
strictatime |
Updates the atime on every read | Maximum |
relatime |
Only if the atime is older than the mtime/ctime, or more than 24 h old | Low (the default since 2009) |
nodiratime |
Like the previous one, but no atime on directories | Low |
noatime |
Never updates the atime | Zero |
relatime is the compromise Linux adopted: it preserves the semantics needed by tools that ask "has this been read since it was last modified?" (mail clients, above all) at a fraction of the cost. noatime removes it altogether.
Meteora mounts /var/lib/meteora with noatime, because no program on the system consults the readings' atime and the saving is real. The general rule: on a server data volume, noatime is almost always right; on a user volume with old mail clients, stay with relatime.
Families of file systems compared
There are dozens. These are the ones you will actually run into, with what really distinguishes them:
| System | Origin | Structure | Journal / CoW | Data checksums | Max. file | When to choose it |
|---|---|---|---|---|---|---|
| ext4 | Linux, 2008 | Inodes + extents | Journal | No (metadata only) | 16 TiB | General Linux server. The safe choice |
| XFS | SGI, 1994 | B+ trees + extents | Journal | No (metadata only) | 8 EiB | Large files, high parallelism, RHEL's default |
| Btrfs | Linux, 2009 | CoW B-trees | Copy-on-write | Yes | 16 EiB | Snapshots, checksums, integrated RAID |
| ZFS | Sun, 2005 | CoW trees + pools | Copy-on-write | Yes | 16 EiB | Serious storage. License outside the Linux kernel |
| F2FS | Samsung, 2012 | Log-structured | Log | Optional | 16 TiB | Flash memory: phones, SD cards |
| NTFS | Microsoft, 1993 | MFT + B-trees | Journal | No | 8 PiB | Windows |
| APFS | Apple, 2017 | CoW trees | Copy-on-write | Metadata only | 8 EiB | macOS, iOS |
| FAT32 | Microsoft, 1996 | FAT table | No | No | 4 GiB | Universal compatibility, UEFI boot |
| exFAT | Microsoft, 2006 | Extended FAT table | No | No | 128 PiB | Large SD cards shared between systems |
| tmpfs | Linux | RAM only | N/A | N/A | RAM + swap | /run, /dev/shm, temporary files |
| NFS / SMB | Network | Client-server | The server's | The server's | The server's | Sharing over the network (04-03) |
Four observations worth more than the whole table:
FAT32 and the 4 GiB limit. It is the real limit most people run into at some point in their lives: copying a 5 GB video to a USB stick fails, and not for lack of space. FAT32 stores the size in 32 bits, so 2³² − 1 = 4,294,967,295 bytes is the absolute ceiling. And it has no journal: a power cut in the middle of a write can leave it inconsistent with no automatic way to repair it. It survives because absolutely everything reads it, including your motherboard's UEFI, which requires a FAT32 partition in order to boot.
tmpfs is not a disk. It lives in the page cache and in swap: blazingly fast and volatile. It is what sits behind /dev/shm/meteora-cache (03-03) and /run/meteora/. That Meteora's cache "is a file" and at the same time lives in RAM is not a contradiction: it is a file in a file system with no device behind it. We will look at it in detail in 04-03.
Copy-on-write versus journal. They are two different philosophies for the same problem — surviving a power cut. The journal writes down what it is about to do before doing it; copy-on-write never overwrites a live block, it writes a new copy and switches the pointer at the end. That is the subject of Space Allocation, Journaling and Integrity.
Data checksums. Only Btrfs and ZFS verify that the data they return is what was written. ext4 and XFS verify their metadata and their journal, but if a bit of your readings gets corrupted on the platter, they hand it to you corrupted without warning. That is silent corruption, and it also belongs to 04-05.
Meteora's decision for /var/lib/meteora
With all of the above, the choice can now be argued rather than copied from a tutorial. Meteora's workload profile:
| Workload characteristic | Value |
|---|---|
| File size | 17.3 MB, one per day |
| Number of files | ~365 a year, about 1,800 in total |
| Write pattern | Append to the end, continuous, ~8 readings/second |
| Read pattern | Full sequential (aggregator) and random via mmap (meteo-api) |
| Device | Software RAID 1 over two NVMe drives (02-05) |
| Integrity requirement | High: the data cannot be regenerated |
| Availability requirement | High: meteo-api serves 24/7 |
The candidates and their reasoned rejection:
| Candidate | Argument in favor | Why it is not chosen |
|---|---|---|
| XFS | Excellent with large files, highly parallel, no inode limit | It cannot be shrunk; the volume might need adjusting |
| Btrfs | Data checksums, native snapshots | Uneven performance with continuous writes; its RAID 5/6 is not reliable |
| ZFS | The best at integrity | Outside the kernel; complicates maintenance and upgrades |
| F2FS | Designed for flash | Aimed at phones; less mileage on servers |
| ext4 | Mature, predictable, extents, tooling | No data checksums |
Decision: ext4, with this justification:
- Maturity and predictability. It is the file system with the most flight hours in the Linux ecosystem. For unrecoverable data, "boring and proven" is worth more than "modern and promising".
- Extents. A contiguous 17 MB file is described with a single extent instead of 4,219 pointers. We will measure it with
filefragin 04-05. - Tooling.
dumpe2fs,debugfs,tune2fs,e2fsckandresize2fs(which can shrink, unlike XFS) make up the most complete set. When something goes wrong at three in the morning, that counts. - The journal in
orderedmode guarantees that another file's data will never appear inside the readings after a failure. We will justify it in 04-05. - Inodes are not an issue for this workload: 1,834 used out of 13 million.
- The lack of data checksums is compensated for by other means:
mdverifies the RAID 1 with a weekly scrub, and theingestorstores a CRC32 per reading in its own format.
The exact formatting parameters:
sudo mkfs.ext4 \
-b 4096 \ # 4 KiB block = page size
-i 1048576 \ # one inode per MiB: few, large files
-m 1 \ # only 1 % reserved for root (default is 5 %)
-L meteora-data \ # stable label
-O extent,dir_index,has_journal,metadata_csum,64bit \
/dev/md0What each option does and why here:
-b 4096: it matches the page and the NVMe's physical sector; it also allows directmmap().-i 1048576: by default there would be 13 million inodes, of which 1,834 are used; with one inode per MiB there are 200,000 left, more than enough, and it saves about 3 GiB of inode table that becomes usable space. It is a safe adjustment only because we know the workload.-m 1: ext4 reserves 5 % for root, so that a full disk does not prevent logging in or operating. On a 200 GiB data volume that means 10 GiB tied up; at 1 % it is 2 GiB, margin enough. Careful: on/never go below 5 %.-O extent: extents instead of indirect pointers (04-05).-O dir_index: hash indexes for large directories (04-02).-O metadata_csum: metadata checksums, which detect corruption of the inode or the bitmap.
And that is how /var/lib/meteora/readings/2026-08-31.dat stops being a magic string and becomes inode 1180934 of the ext4 labeled meteora-data on /dev/md0.
Common Mistakes and Tips
Believing that ctime is the creation date. It is change time: it changes with chmod, chown or when a hard link is created. The real creation time is the crtime, which only stat shows on modern systems. Confusing them leads to false conclusions in any forensic analysis.
Misreading stat's Blocks field. They are always 512-byte blocks, even if the file system uses 4 KiB. If you multiply by 4,096 you will get a size eight times larger than the real one.
Not looking at df -i when faced with an ENOSPC. A disk at 26 % that reports "No space left on device" is almost always inode exhaustion. Always check both.
Assuming the number of inodes can be increased. In ext4 it is fixed at format time and there is no going back. If you foresee millions of small files, decide -i beforehand, or use XFS or Btrfs, which allocate them dynamically.
Choosing a large block "because it is faster". With a million 800-byte files, a 64 KiB block wastes 99 % of the space. The block size depends on the distribution of your files' sizes, not on an intuition about speed. And on Linux it cannot exceed the page's 4 KiB.
Running mkfs on the wrong partition. mkfs does not ask. Always confirm with lsblk and blkid before pressing Enter; we will see this in 04-03.
Tip: save the dumpe2fs -h output of your volumes. Having the UUID, the block size and the positions of the backup superblocks at hand turns an emergency recovery into a two-minute formality.
Tip: use stat instead of ls -l when something does not add up. It shows the four dates, the inode, the device and the real blocks. Half the mysteries involving files are solved by reading a stat calmly.
Tip: check the type before assuming. Before running cat on something unknown, look at the first character of ls -l. A cat on a 200 GiB block device, or on a FIFO with no writer, does not end the way you expect.
Exercises
Exercise 1: identifying types and reading a stat
Without using ls -l, write a command that classifies everything in /dev, /run and your home directory by file type, counting how many there are of each. Then take a regular file of at least 10 MB, run stat on it, and answer: (a) how many 4 KiB blocks does it actually occupy? (b) how much space is wasted in the last block? (c) do mtime and ctime match, and what does it mean if they do or do not?
Exercise 2: the block size trade-off
A 500 GiB volume is going to host one of two possible workloads. Workload A: Meteora's daily files, 17,280,000 bytes each, for 30 years. Workload B: 20 million 600-byte session files. For block sizes of 1, 4 and 64 KiB, calculate for each workload: total blocks needed, space actually occupied, absolute waste and percentage. Then state which size you would choose for each workload and whether a single volume can serve both.
Exercise 3: diagnosing a misleading ENOSPC
A colleague warns you: the session service on meteo-01 is failing with "No space left on device", but df -h shows the volume at 31 %. Write the complete diagnostic procedure — which commands you run, in what order and what you expect to see in each one — the technical explanation of why it happens, the immediate fix to restore the service and the definitive fix so it does not happen again. Include the formatting command you would use and justify its parameters.
Solutions
Solution 1
Classifying by type is done with find -type or, better, with stat's format:
for d in /dev /run "$HOME"; do
echo "=== $d ==="
find "$d" -maxdepth 1 -printf '%y\n' 2>/dev/null | sort | uniq -c | sort -rn
done-printf '%y' prints a single letter with the type (f regular, d directory, l link, b, c, p, s), and sort | uniq -c counts them. Typical output:
=== /dev ===
198 c ← character devices: terminals, /dev/null, /dev/random
42 b ← block devices: disks and partitions
28 d
18 l
=== /run ===
34 d
12 s ← service sockets, including /run/meteora/api.sock
6 p ← FIFOs, including /run/meteora/readings.fifoFor the questions, with a stat giving Size: 17280000 and Blocks: 33760:
(a) stat counts 512 B blocks: 33,760 × 512 = 17,285,120 bytes. In 4 KiB blocks that is 17,285,120 / 4,096 = 4,220 blocks. The data needs ⌈17,280,000 / 4,096⌉ = 4,219, so the extra block is extent tree metadata.
(b) 17,280,000 = 4,218 complete blocks + 3,072 bytes. The last block uses 3,072 out of 4,096, so 1,024 bytes are wasted: 0.006 % of the file. Irrelevant here, decisive with 800-byte files.
(c) If mtime and ctime match, the last operation was a content write (which updates both). If the ctime is later, some piece of metadata was modified after the write: a chmod, a chown or an ln. A ctime later than the mtime with no known cause is exactly what makes an analyst suspicious.
Solution 2
Workload A — one file of 17,280,000 B, 30 years × 365 = 10,950 files:
| Block | Blocks/file | Occupied/file | Waste/file | Total waste |
|---|---|---|---|---|
| 1 KiB | 16,875 | 17,280,000 | 0 | 0 |
| 4 KiB | 4,219 | 17,281,024 | 1,024 | 11.2 MB |
| 64 KiB | 264 | 17,301,504 | 21,504 | 235 MB |
Over 189 GB of data, even 235 MB is 0.12 %. With 4 KiB, the waste is 0.006 %: negligible. What rules here is the number of blocks to manage, and 4,219 is far easier to handle than 16,875.
Workload B — 20,000,000 files of 600 B (12 GB of real data):
| Block | Occupied/file | Total occupied | Waste | % wasted |
|---|---|---|---|---|
| 1 KiB | 1,024 | 20.5 GB | 8.5 GB | 41 % |
| 4 KiB | 4,096 | 81.9 GB | 69.9 GB | 85 % |
| 64 KiB | 65,536 | 1,310 GB | 1,298 GB | 99.1 % |
With 64 KiB it does not fit: it would take 1.3 TB on a 500 GiB volume to store 12 GB of data.
Choice. Workload A: 4 KiB, for its affinity with the page and mmap(), with effectively zero waste. Workload B: 1 KiB, which saves 61 GB compared with 4 KiB. And an important warning for workload B: 20 million files need 20 million inodes, and the default mkfs would give 32.7 million — only just — so you have to set -i 2048 or use XFS.
A single volume for both? Technically yes, but it is a bad idea: the requirements are opposites (large block and few inodes versus small block and many inodes), and besides, mixing workloads means the millions of small files fragment the space the large ones need. Two separate volumes, which is exactly the partitioning argument we will see in 04-03.
Solution 3
Diagnostic procedure, in order:
df -h /var/spool/cache # 1. space: 31 % → not this
df -i /var/spool/cache # 2. inodes: 100 % → HERE IT IS
findmnt /var/spool/cache # 3. confirm the device and the options
sudo find /var/spool/cache -xdev -type f -printf '%h\n' \
| sort | uniq -c | sort -rn | head # 4. who created them
sudo dumpe2fs -h /dev/sdb1 | grep -E 'Inode count|Free inodes|Inode size'Step 2 is the diagnosis and step 4 identifies the culprit. -xdev matters: without it, find would cross into other file systems and count files that do not consume this volume's inodes.
Technical explanation. In ext4 the number of inodes is fixed and decided at format time, with one inode per 16 KiB by default. A 50 GiB volume therefore has 3,276,800 inodes. Every file, however small, consumes one. When they run out, creat() and open(O_CREAT) return ENOSPC — "No space left on device" — even though free blocks remain, because the kernel does not distinguish between the two resources in the error code. The 600-byte session files take one 4 KiB block and one inode each: the inodes run out once you reach 3.2 million, with only 13 GB used.
Immediate fix (restore the service):
sudo find /var/spool/cache/sessions -type f -mtime +7 -delete
df -i /var/spool/cache # verify that there are free inodes now
sudo systemctl restart sessionsDeleting files older than 7 days frees inodes immediately. With millions of files it is better to use find ... -delete instead of rm -rf, which can overflow the command line.
Definitive fix, in three layers:
- Automatic purge: a
systemdunit with a timer (module 7) or atmpfiles.dentry that deletes expired sessions every hour. - Reformat with the right density, after taking a backup:
-b 1024 because the files are 600 bytes and with 4 KiB you waste 85 %; -i 2048 gives one inode per 2 KiB, that is 26 million inodes, far above the original 3.2; -m 0 because it is a temporary data volume where no root reserve is needed.
- Monitoring: alert when
df -igoes above 80 %, exactly the way space is watched. Almost no monitoring system does it by default, and that is why this failure keeps showing up.
A reasonable alternative: mount /var/spool/cache as tmpfs, since sessions are volatile by nature. Both problems disappear at once, at the cost of RAM and of losing them on reboot (04-03).
Conclusion
A file is a named sequence of bytes, with a size and an owner, that the system places wherever it likes and the program reads as if it were continuous. That abstraction solves in one stroke the seven problems any application would face working on module 2's raw block vector, and it does so with the same strategy as virtual memory: a translation table that turns an orderly logical space into a scattered physical one.
A file has three parts in three places: the content in the data area, the metadata in the inode, and the name in the directory. The central consequence, from which half this module hangs, is that the name is not inside the file: inode 1180934 does not know it is called 2026-08-31.dat.
In UNIX there are seven types of file — regular, directory, symbolic link, block device, character device, FIFO and socket — and the last four take up an inode but zero data blocks: the FIFO /run/meteora/readings.fifo and the api.sock socket from module 3 are file system entries whose content lives in the kernel. ls -l tells them apart by their first character and stat reports everything: logical size, 512-byte blocks, device, inode, link count and the four dates.
Physically, an ext4 is divided into block groups of 128 MiB, each with a block bitmap, an inode bitmap, an inode table and a data area, preceded by the superblock — the identity card, with backup copies that save volumes — and the group descriptors. Going from an inode number to its position on disk costs two divisions and one multiplication, and that is the deep reason why the inode is a number.
The block size is a measurable trade-off: 4 KiB is the standard because it matches the memory page (and therefore the page cache and mmap()) and the modern physical sector; going lower makes sense with millions of tiny files, and going higher is not even possible on Linux. Inodes, in ext4, are fixed at format time: hence the ENOSPC with a half-empty disk that only df -i explains. And the four timestamps distinguish reading (atime), modifying content (mtime), modifying metadata (ctime) and creating (crtime); noatime removes 48,000 useless daily writes on meteo-01.
With the comparison of families in hand, Meteora chooses ext4 for /var/lib/meteora: maturity in the face of unrecoverable data, extents to describe 17 MB in a single run, the most complete toolset for three in the morning, and a deliberate tuning of -i 1048576 and -m 1 that recovers about 11 GiB for useful data.
We are left with the loose end we have been dropping in every section: the name. We know it is not in the inode, we know it lives "in the directory", and we have said that a directory is "a special file", but we have not opened one. What exactly is inside /var/lib/meteora/readings/? How does the system manage to turn the string /var/lib/meteora/readings/2026-08-31.dat into the number 1180934, and how many disk accesses does that cost? Why can a symbolic link end up broken and a hard one cannot? And what happens when a directory has a hundred thousand files inside it?
All of that is what the next lesson opens: Directory Structures.
Operating Systems Fundamentals
Module 1: Introduction to Operating Systems
- Basic Concepts of Operating Systems
- History and Evolution of Operating Systems
- Types of Operating Systems
- Main Functions of an Operating System
- Kernel Architecture: Monolithic, Microkernel and Hybrid
- User Mode, Kernel Mode and System Calls
Module 2: Resource Management
- Process Management
- CPU Scheduling
- Memory Management
- Virtual Memory and Paging
- Storage Management
- Device Management
- Drivers, Interrupts and I/O Operations
Module 3: Concurrency
- Concurrency Concepts
- Threads and Processes
- Inter-Process Communication (IPC)
- Synchronization and Mutual Exclusion
- Classic Concurrency Problems
- Deadlocks: Prevention, Detection and Recovery
Module 4: File Structures
- File Systems
- Directory Structures
- Partitions, Mounting and the Virtual File System
- File Management
- Space Allocation, Journaling and Integrity
- File Security and Permissions
Module 5: System Protection and Security
- Protection Principles and Access Control
- Users, Authentication and Privilege Escalation
- Common Threats and System Hardening
- Auditing, Logging and Incident Response
Module 6: Virtualization and Containers
- Virtualization: Hypervisors and Virtual Machines
- Containers: Namespaces and cgroups
- The Operating System in the Cloud
- Mobile and Real-Time Operating Systems
