We closed module 3 with an uncomfortable admission: for three modules we had been writing /var/lib/meteora/readings/2026-08-31.dat as if that string of text were something self-evident, when in module 2 we left storage in a very different place. There, an NVMe SSD was a vector of blocks numbered 0 to N, addressable by LBA, with no names, no folders, no sizes and no owners. A giant array of 512 or 4,096 bytes per slot. Nothing more.

Between that vector of blocks and the path /var/lib/meteora/readings/2026-08-31.dat sits one of the most successful abstractions in computing: the file system. It is the piece that turns "block 8,394,271" into "yesterday's readings file", that remembers who created it and when, that knows it takes up 17 MB spread over thousands of scattered blocks, and that lets a C program read it without knowing anything at all about LBAs, sectors or geometry.

This lesson builds that abstraction from the ground up. We are going to define precisely what a file is, see what metadata it carries and where that metadata lives, dissect the inode — the central structure of every UNIX system — walk through the physical layout of a file system on the device, calculate why the block size is a trade-off and not an accident, and finish by comparing the families of file systems that exist so we can decide, with arguments, which one deserves /var/lib/meteora.

Contents

  1. From the raw block to the file: what problem the abstraction solves
  2. What a file is: content, metadata and name
  3. The seven UNIX file types
  4. ls -l and stat interpreted field by field
  5. The inode: what it contains and what it does not
  6. Inode numbers, ls -i and exhaustion with df -i
  7. Physical layout of a file system on the device
  8. Block size and its trade-off, calculated
  9. Timestamps: atime, mtime, ctime, crtime and the reason for noatime
  10. Families of file systems compared
  11. Meteora's decision for /var/lib/meteora

From the raw block to the file: what problem the abstraction solves

Imagine for a moment that Meteora had no file system and worked directly on /dev/nvme0n1, the block device from module 2. The ingestor would have to solve, all by itself, this list of problems:

Problem What the ingestor would have to do without a file system
Location Remember which LBA today's data starts at, and store that somewhere... where?
Growth Know how many blocks to reserve in advance, because it cannot "grow" over a neighbour
Free space Keep its own accounting of which blocks are used and which are not
Names Invent its own index translating "31 August" into a block number
Concurrency Coordinate with the aggregator and with meteo-api so they do not step on each other's blocks
Permissions There are none: anyone with access to the device reads and writes everything
Persistence of the accounting If the power fails while it updates its index, it loses everything

Every program on the server would have to solve all seven, each one its own way, and none could cooperate with the others. It is exactly the situation we described in 01-01 when we talked about the operating system as an extended machine: without it, every application reimplements the hardware.

The file system solves all seven at once with a single idea:

A file is a named sequence of bytes, with a size and an owner, that the system stores wherever it likes and the program reads as if it were continuous.

The four key phrases in that definition deserve attention:

  • Sequence of bytes. In UNIX a file has no internal structure known to the system. There are no "records", no "fields", no types. That 2026-08-31.dat contains 720,000 24-byte Reading structs is an agreement among Meteora's own programs, invisible to the kernel. Other historical systems (IBM's, VMS) did impose a record structure, and the industry eventually decided UNIX's flat model was right, because of its simplicity.
  • Named. The name is the handle by which the user grabs the file. In section 5 we will see the surprise: the name is not inside the file.
  • Wherever it likes. The system decides which blocks to use. That is what we will study in Space Allocation, Journaling and Integrity.
  • As if it were continuous. This is the magic. 2026-08-31.dat occupies 4,219 blocks that may be scattered all over the SSD, and yet read() delivers them in order as a stream of bytes. It is the same class of illusion as the virtual memory of module 2: contiguous logical addresses over scattered physical storage.

In fact the parallel with virtual memory is so exact that it is worth pinning down, because it will save you effort throughout the module:

Virtual memory (02-04) File system (module 4)
The process's virtual address space The file as a sequence of bytes 0..N
Page (4 KiB) Logical block (4 KiB)
Page frame in RAM Physical block on the device
Page table Inode pointers/extents
The MMU translates virtual → physical The file system translates offset → LBA
A page fault brings the page in from disk read() brings the block into the page cache

It is the same idea applied twice: a translation table that turns an orderly logical space into a disorderly physical one.

What a file is: content, metadata and name

A file has three parts, and they live in three different places. This separation is the key to everything else:

graph LR
    subgraph DIR["Directory (04-02)"]
        N["name: 2026-08-31.dat<br/>inode: 1180934"]
    end
    subgraph INO["Inode no. 1180934"]
        M["type, permissions, owner,<br/>size, dates, pointers"]
    end
    subgraph DAT["Data area"]
        D["4,219 blocks<br/>holding the 17,280,000 bytes"]
    end
    N -->|points to| M
    M -->|points to| D
  • The content: the bytes themselves, in the device's data area.
  • The metadata: everything the system knows about the file. It lives in the inode.
  • The name: it lives in the directory that contains it, not in the file. We will develop this in Directory Structures, but you need to know it already to understand the inode.

A file's typical metadata, with what each item means:

Metadata What it is Example in Meteora
Type Regular, directory, symbolic link... Regular
Size Bytes of content 17,280,000
Blocks 512 B blocks actually occupied 33,760
Owner (UID) Owning user meteora (UID 990)
Group (GID) Owning group meteora (GID 990)
Permissions 12 mode bits 0640
Link count How many names point to this inode 1
Timestamps Access, modification, change, creation see section 9
Data pointers Where the blocks are extents (04-05)

Notice something you can already sense: the name does not appear in that list. It is not an oversight; it is UNIX's central design decision, and out of it come hard links, deferred deletion and half a dozen behaviors that are surprising until you understand this.

The seven UNIX file types

In UNIX, the phrase "everything is a file" is meant seriously. The same interface — open, read, write, close — works for a data file, a keyboard, a pipe or a network connection. What changes is the type, a 4-bit field in the inode:

Symbol in ls -l Type What it is Meteora example
- Regular Sequence of bytes on disk /var/lib/meteora/readings/2026-08-31.dat
d Directory Table of (name, inode) pairs /var/lib/meteora/readings/
l Symbolic link A file containing a path /var/lib/meteora/readings/today.dat
b Block device Block access, cached /dev/nvme0n1, /dev/md0
c Character device Byte access, uncached /dev/null, /dev/random, /dev/tty
p FIFO (named pipe) A channel in the file system /run/meteora/readings.fifo
s Socket Local communication endpoint /run/meteora/api.sock

The last three are old friends from module 3: we used the FIFO at /run/meteora/readings.fifo in Inter-Process Communication (IPC), and UNIX domain sockets too. The difference is that there we saw them as IPC mechanisms and here we see them as file system entries: they have an inode, permissions and a name, but their content is not on disk — a FIFO has a buffer in kernel memory, a socket has a network queue. They take up one inode and zero data blocks.

The same goes for block and character devices, which we saw in Device Management: their inode holds no data pointers, it holds the (major number, minor number) pair that identifies the driver. A device file is, literally, an inode with a couple of integers inside.

ls -l and stat interpreted field by field

Let us really look at it. This is the listing of Meteora's directory on meteo-01:

$ ls -l /var/lib/meteora/readings/ /dev/nvme0n1 /dev/null /run/meteora/
-rw-r----- 1 meteora meteora 17280000 Aug 31 23:59 2026-08-31.dat
-rw-r----- 1 meteora meteora 17280000 Sep  1 12:40 2026-09-01.dat
lrwxrwxrwx 1 meteora meteora       14 Sep  1 00:00 today.dat -> 2026-09-01.dat
drwxr-x--- 2 meteora meteora     4096 Sep  1 00:00 archive

brw-rw---- 1 root  disk    259, 0 Sep  1 08:12 /dev/nvme0n1
crw-rw-rw- 1 root  root      1, 3 Sep  1 08:12 /dev/null

prw-r----- 1 meteora meteora      0 Sep  1 08:13 /run/meteora/readings.fifo
srwxr-xr-x 1 meteora meteora      0 Sep  1 08:13 /run/meteora/api.sock

The fields, from left to right:

  1. First character: the type from the previous table. -, l, d, b, c, p, s. At a glance you already know what each thing is.
  2. The next nine characters: the permissions, which we will look at in File Security and Permissions.
  3. The number after the permissions: the link count. It is 1 for ordinary files and 2 for the archive directory (04-02 explains why).
  4. Owning user and group: meteora meteora.
  5. Size: 17,280,000 bytes for the readings. But look at /dev/nvme0n1: where the size ought to be it says 259, 0. Those are the major and the minor. A device file has no size because it has no content; in its place ls shows the pair that identifies the driver. And the FIFO and the socket show 0, for the same reason.
  6. Date: by default the mtime, not the creation time nor the last access time (section 9).
  7. Name, with the -> target on the symbolic link.

ls -l is a summary. To see everything you use stat:

$ stat /var/lib/meteora/readings/2026-08-31.dat
  File: /var/lib/meteora/readings/2026-08-31.dat
  Size: 17280000    Blocks: 33760      IO Block: 4096   regular file
Device: 9,0         Inode: 1180934     Links: 1
Access: (0640/-rw-r-----)  Uid: (  990/meteora)   Gid: (  990/meteora)
Access: 2026-09-01 06:00:11.482913711 +0200
Modify: 2026-08-31 23:59:58.117204339 +0200
Change: 2026-08-31 23:59:58.117204339 +0200
 Birth: 2026-08-31 00:00:00.004118220 +0200

Field by field, with what you need to understand about each one:

  • Size: 17280000. The file's logical bytes: exactly 720,000 readings × 24 bytes. It is the number lseek(fd, 0, SEEK_END) returns.
  • Blocks: 33760. Here is a classic trap: stat counts 512-byte blocks, always, regardless of the file system's real block size. 33,760 × 512 = 17,285,120 bytes occupied on disk, somewhat more than the 17,280,000 logical ones. The difference is 5,120 bytes: 4 KiB for the partially used last block plus the metadata of the extent tree. When "blocks × 512" is smaller than the size, that is the signature of a sparse file (we will see it in 04-04).
  • IO Block: 4096. The preferred block size for reads; reading in multiples of this number saves the kernel extra work.
  • Device: 9,0. Major 9, minor 0: /dev/md0, the RAID 1 we assembled in 02-05. This pair identifies which file system the inode lives in, and the (device, inode) pair is the only thing that uniquely identifies a file across the whole machine.
  • Inode: 1180934. The inode number. Section 6 is devoted to it.
  • Links: 1. A single name points to this inode (04-02).
  • Access (0640). Twelve mode bits, shown in octal and symbolically (04-06).
  • Uid/Gid 990. The meteora user and group. Careful: the inode stores numbers, not names; the translation into meteora is done by stat consulting /etc/passwd.
  • The four dates: section 9.

The inode: what it contains and what it does not

The inode (index node) is the data structure that is the file. Everything else is a reference to it. It lives on disk, in a reserved area, and has a fixed size: 256 bytes on a modern ext4 (128 on old ones, configurable at format time).

Its contents, grouped:

Group Fields Approx. bytes
Identity Type (4 bits) + permissions (12 bits), UID, GID 8
Size Logical size in bytes (64 bits), blocks occupied 12
Links Count of names referencing it 2
Times atime, mtime, ctime, dtime, crtime (with nanoseconds) 40
Data 60 bytes of pointers/extents (04-05) 60
Extras Flags (chattr), version, checksum, extended attributes the rest

And now the important part, which is what is not there:

The file's name is not in the inode. Neither is the path, nor the directory it belongs to, nor anything relating it to /var/lib/meteora/readings/2026-08-31.dat.

Inode 1180934 does not know what it is called. It knows it is a regular file of 17,280,000 bytes belonging to UID 990, that there is 1 name somewhere pointing at it, and which blocks hold its data. Nothing else.

This is not a limitation: it is the design decision half this module hangs from. Its consequences, which we will unfold as we go:

Consequence Where it is explained
One and the same file can have several names (hard links) 04-02
Renaming is instantaneous: only a directory entry changes 04-02
Deleting is unlink: removing one name, not destroying the file 04-02
An open file with no name at all still exists 04-02, 04-04
The name has no permissions; the permissions are in the inode 04-06
A file in use can be replaced atomically with rename() 04-04

Keep this sentence: the inode is the file; the name is just a label stuck on from outside.

Inode numbers, ls -i and exhaustion with df -i

Every inode has a unique number within its file system. You can see it with ls -i:

$ ls -i /var/lib/meteora/readings/
1180934 2026-08-31.dat   1180935 2026-09-01.dat   1180936 today.dat

Two files in the same file system with the same inode number are the same file. Two files in different file systems may share a number without having anything to do with each other: that is why the real identity is the (device, inode) pair, st_dev + st_ino in stat. It is exactly what find -samefile checks and what rsync uses so it does not copy the same linked content twice.

Now the practical part. In ext4, the number of inodes is fixed at format time and cannot be increased afterwards. They are reserved in advance, with a default ratio of one inode per 16 KiB of capacity:

$ df -h /var/lib/meteora
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        196G   41G  146G  22% /var/lib/meteora

$ df -i /var/lib/meteora
Filesystem        Inodes  IUsed     IFree IUse% Mounted on
/dev/md0        13107200   1834  13105366    1% /var/lib/meteora

Interpretation: the volume has 13,107,200 inodes (200 GiB ÷ 16 KiB) and uses only 1,834, because Meteora stores one large file per day and has been running for about five years. With that policy, in 100 years it would use 36,500 inodes: 0.28 %.

The other way round, however, produces the most bewildering failure in systems administration:

$ df -h /var/spool/cache
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1        50G   12G   36G  26% /var/spool/cache      ← 74 % free!

$ df -i /var/spool/cache
Filesystem       Inodes   IUsed   IFree IUse% Mounted on
/dev/sdb1       3276800 3276800       0  100% /var/spool/cache ← 0 free

$ touch /var/spool/cache/test
touch: cannot touch 'test': No space left on device

"No space left on device" with 36 GB free. The error is ENOSPC and it is literal from the kernel's point of view: there is no inode space left. A process that creates millions of tiny files — caches, sessions, mail queues — exhausts the inode reserve long before the blocks. Diagnostic rule: faced with an ENOSPC, always run both df -h and df -i; if the first explains nothing, the second will.

To find the culprit:

$ sudo find /var/spool/cache -xdev -type f -printf '%h\n' | sort | uniq -c | sort -rn | head -5
2984112 /var/spool/cache/sessions
  12043 /var/spool/cache/tmp

The command counts files per directory: -xdev avoids crossing into other file systems, -printf '%h\n' prints each file's directory, and sort | uniq -c | sort -rn groups and sorts. Almost three million files in sessions: there is the problem.

The fix is not to enlarge the disk, because inodes do not grow: you have to delete files or reformat with mkfs.ext4 -i 4096 (one inode per 4 KiB, four times as many) or -N 8000000 (an explicit number). This is one of the arguments in favor of XFS and Btrfs, which allocate inodes dynamically and do not suffer from this problem.

Physical layout of a file system on the device

We now know what an inode is. Where is it physically? Let us open the device up. The hierarchy of units, from smallest to largest:

Unit Typical size Who defines it
Sector 512 B (logical) / 4,096 B (physical) The disk hardware
Logical block 1, 2 or 4 KiB The file system, at format time
Block group 128 MiB in ext4 The file system
File system The whole partition The administrator

The sector is the smallest unit the device knows how to read or write; we saw it in 02-05. The logical block is the smallest unit the file system knows how to allocate: even though the disk can read 512 bytes, ext4 never allocates less than one block to a file.

An ext4 is divided into block groups of 128 MiB, each with its own accounting. The reason is performance and comes straight out of module 2: keeping together the metadata and the data that are used together reduces head movement on an HDD and improves locality on an SSD.

graph TB
    subgraph FS["/dev/md0 — 200 GiB ext4, 1,600 groups of 128 MiB"]
        BOOT["Block 0<br/>1 KiB of boot area"]
        subgraph G0["Block group 0"]
            SB["Superblock<br/>(1 block)"]
            GD["Group descriptors<br/>(N blocks)"]
            BB["BLOCK<br/>bitmap<br/>(1 block)"]
            IB["INODE<br/>bitmap<br/>(1 block)"]
            IT["Inode table<br/>(512 blocks)"]
            DZ["DATA AREA<br/>(~31,000 blocks)"]
        end
        G1["Group 1<br/>(same structure)"]
        GN["... Group 1,599"]
    end
    BOOT --> SB --> GD --> BB --> IB --> IT --> DZ --> G1 --> GN

Piece by piece:

The superblock. It is the file system's identity card, and it takes up a single block. It contains the total number of inodes and blocks, how many are free, the block size, the inode size, the number of blocks per group, the UUID, the label, the date of the last mount, the mount counter and the state (clean or dirty, which will be decisive in 04-05). Without a superblock nothing can be mounted, because you do not even know where the inodes begin. That is why ext4 keeps backup copies in several groups:

$ sudo dumpe2fs /dev/md0 | grep -i superblock
Primary superblock at 0, Group descriptors at 1-13
  Backup superblock at 32768, Group descriptors at 32769-32781
  Backup superblock at 98304, Group descriptors at 98305-98317
  Backup superblock at 163840, ...

If the primary one is corrupted, you recover with sudo e2fsck -b 32768 /dev/md0, which tells e2fsck to use the backup at block 32768. It is a command worth writing down: it saves volumes that looked lost.

The group descriptors. An array with one entry per group, saying where that group's bitmaps and inode table are and how many free elements it has. It is the index that lets you find everything else.

The block bitmap. One bit per block in the group: 1 = used, 0 = free. With 4 KiB blocks, a 128 MiB group has 32,768 blocks, which is 32,768 bits = 4,096 bytes: exactly one block. That is no coincidence; the group size is chosen precisely so that its bitmap fits in one block.

The inode bitmap. The same thing for the group's inodes.

The inode table. The array of inodes proper. If a group has 8,192 inodes of 256 bytes, it takes up 2 MiB = 512 blocks. This is where an inode's position is computed from its number, with pure arithmetic and without searching any index:

/* Locate inode 1180934 in an ext4 with 8,192 inodes per group */
unsigned group    = (1180934 - 1) / 8192;      /* = 144  → group 144        */
unsigned index    = (1180934 - 1) % 8192;      /* = 1157 → position 1157    */
off_t    position = inode_table_start[144] + (off_t)1157 * 256;

Two divisions and one multiplication: that is all it costs to go from an inode number to its position on disk. Inodes are numbered from 1, hence the - 1. This is the deep reason why an inode is a number and not a name: looking up a name would require walking a table; a number is resolved with arithmetic in nanoseconds.

The data area. Everything else: the blocks holding the files' bytes.

You can see it on your own machine with dumpe2fs:

$ sudo dumpe2fs -h /dev/md0 | head -20
Filesystem volume name:   meteora-data
Filesystem UUID:          9f3a1c22-7d4e-4a51-b7c8-2e5f0a1d6b93
Filesystem features:      has_journal ext_attr dir_index extent 64bit
Filesystem state:         clean
Inode count:              13107200
Block count:              52428800
Free blocks:              41156923
First block:              0
Block size:               4096
Blocks per group:         32768
Inodes per group:         8192
Inode size:               256

Every line is a real superblock field. Block count: 52428800 × 4,096 = 200 GiB. Filesystem state: clean means it was unmounted properly. And dir_index and extent are features that will show up in 04-02 and 04-05.

Block size and its trade-off, calculated

The block size is chosen at format time and cannot be changed afterwards. It is a genuine trade-off, and it shows up in the numbers.

The cost of a large block: internal fragmentation. Since allocation happens in whole blocks, each file's last block is left partially empty, and that space is lost. On average half a block per file is wasted.

The cost of a small block: more blocks to manage. More entries in the metadata, more allocation work, and smaller I/O requests.

With Meteora's real files (17,280,000 bytes):

Block size Blocks needed Space occupied Waste Bitmap for 200 GiB
1 KiB 16,875 17,280,000 B 0 B (exact) 25 MiB
2 KiB 8,438 17,281,024 B 1,024 B 12.5 MiB
4 KiB 4,219 17,281,024 B 1,024 B 6.25 MiB
16 KiB 1,055 17,285,120 B 5,120 B 1.6 MiB
64 KiB 264 17,301,504 B 21,504 B 0.4 MiB

For 17 MB files the waste is irrelevant in every case: 21 KB out of 17 MB is 0.12 %. But the number of blocks to manage changes by a factor of 64.

Now the opposite scenario, a directory with a million 800-byte files (sessions, not readings):

Block size Space occupied by 1,000,000 files Waste
1 KiB 1,024 MB 224 MB (22 %)
4 KiB 4,096 MB 3,296 MB (80 %)
64 KiB 65,536 MB 64,736 MB (99 %)

With 64 KiB blocks, a million 800-byte files would occupy 64 GB to store 800 MB. The very same block size that was irrelevant for the readings is catastrophic here.

Why 4 KiB is the universal standard, and not an arbitrary convention:

  1. It matches the page size (module 2). The kernel's page cache works in 4 KiB pages; if the block is the same size, one block = one page and nothing has to be split or joined. A block larger than the page cannot be mapped directly with mmap(), which is precisely what the aggregator does with 2026-08-31.dat (02-04). In fact, Linux does not support blocks larger than the page size, so on x86-64 the maximum is 4 KiB.
  2. It matches the physical sector of modern disks (Advanced Format, 4Kn), so no write triggers the read-modify-write cycle.
  3. It balances the waste for the real distribution of file sizes on a typical system.

Practical rule: use 4 KiB unless you have a measured reason not to. The reason to go down to 1 KiB is a volume dedicated to millions of tiny files; to go up you would have to move to XFS on architectures with larger pages, or to file systems with bigalloc.

Timestamps: atime, mtime, ctime, crtime and the reason for noatime

Four dates, and three of them get confused constantly:

Timestamp Name Updated when... Seen with
atime access time The content is read ls -lu, stat
mtime modify time The content changes ls -l (by default)
ctime change time The inode changes ls -lc, stat
crtime creation time The file was created stat (ext4, XFS, Btrfs)

The most frequent mistake is believing that ctime is creation time. It is not: it is change time, the moment any piece of metadata changed. Changing the permissions with chmod, the owner with chown or creating a hard link modifies the ctime but not the mtime, because the content has not changed. And since it cannot be set backwards (not even touch -d touches it), the ctime is the timestamp a forensic analyst looks at: whoever tampers with a file can fake atime and mtime, but in doing so they update the ctime, giving themselves away. We will come back to it in module 5.

A table of what each operation touches on 2026-08-31.dat:

Operation atime mtime ctime
cat 2026-08-31.dat Yes No No
echo data >> 2026-08-31.dat No Yes Yes
chmod 640 2026-08-31.dat No No Yes
chown meteora: 2026-08-31.dat No No Yes
mv 2026-08-31.dat old.dat No No No¹
ln 2026-08-31.dat copy.dat No No Yes²

¹ Renaming changes the directory, not the file's inode. ² It changes the link count, which is in the inode.

Why noatime improves performance

Here is the practical problem. Updating the atime means writing to disk when you read a file. It sounds absurd, and it is: it turns a read-only operation into a read-and-write one.

The numbers for Meteora. meteo-api serves about 2,000 queries an hour, each one reading the day's file. With strict atime, that is 2,000 writes an hour to the inode of a file nobody has modified: 48,000 daily metadata writes, plus their corresponding journal entry (04-05), on a RAID 1 where every write goes to both disks. And on an SSD, write cycles spent for nothing.

The available mount options, which we will apply in Partitions, Mounting and the Virtual File System:

Option Behavior Cost
strictatime Updates the atime on every read Maximum
relatime Only if the atime is older than the mtime/ctime, or more than 24 h old Low (the default since 2009)
nodiratime Like the previous one, but no atime on directories Low
noatime Never updates the atime Zero

relatime is the compromise Linux adopted: it preserves the semantics needed by tools that ask "has this been read since it was last modified?" (mail clients, above all) at a fraction of the cost. noatime removes it altogether.

Meteora mounts /var/lib/meteora with noatime, because no program on the system consults the readings' atime and the saving is real. The general rule: on a server data volume, noatime is almost always right; on a user volume with old mail clients, stay with relatime.

Families of file systems compared

There are dozens. These are the ones you will actually run into, with what really distinguishes them:

System Origin Structure Journal / CoW Data checksums Max. file When to choose it
ext4 Linux, 2008 Inodes + extents Journal No (metadata only) 16 TiB General Linux server. The safe choice
XFS SGI, 1994 B+ trees + extents Journal No (metadata only) 8 EiB Large files, high parallelism, RHEL's default
Btrfs Linux, 2009 CoW B-trees Copy-on-write Yes 16 EiB Snapshots, checksums, integrated RAID
ZFS Sun, 2005 CoW trees + pools Copy-on-write Yes 16 EiB Serious storage. License outside the Linux kernel
F2FS Samsung, 2012 Log-structured Log Optional 16 TiB Flash memory: phones, SD cards
NTFS Microsoft, 1993 MFT + B-trees Journal No 8 PiB Windows
APFS Apple, 2017 CoW trees Copy-on-write Metadata only 8 EiB macOS, iOS
FAT32 Microsoft, 1996 FAT table No No 4 GiB Universal compatibility, UEFI boot
exFAT Microsoft, 2006 Extended FAT table No No 128 PiB Large SD cards shared between systems
tmpfs Linux RAM only N/A N/A RAM + swap /run, /dev/shm, temporary files
NFS / SMB Network Client-server The server's The server's The server's Sharing over the network (04-03)

Four observations worth more than the whole table:

FAT32 and the 4 GiB limit. It is the real limit most people run into at some point in their lives: copying a 5 GB video to a USB stick fails, and not for lack of space. FAT32 stores the size in 32 bits, so 2³² − 1 = 4,294,967,295 bytes is the absolute ceiling. And it has no journal: a power cut in the middle of a write can leave it inconsistent with no automatic way to repair it. It survives because absolutely everything reads it, including your motherboard's UEFI, which requires a FAT32 partition in order to boot.

tmpfs is not a disk. It lives in the page cache and in swap: blazingly fast and volatile. It is what sits behind /dev/shm/meteora-cache (03-03) and /run/meteora/. That Meteora's cache "is a file" and at the same time lives in RAM is not a contradiction: it is a file in a file system with no device behind it. We will look at it in detail in 04-03.

Copy-on-write versus journal. They are two different philosophies for the same problem — surviving a power cut. The journal writes down what it is about to do before doing it; copy-on-write never overwrites a live block, it writes a new copy and switches the pointer at the end. That is the subject of Space Allocation, Journaling and Integrity.

Data checksums. Only Btrfs and ZFS verify that the data they return is what was written. ext4 and XFS verify their metadata and their journal, but if a bit of your readings gets corrupted on the platter, they hand it to you corrupted without warning. That is silent corruption, and it also belongs to 04-05.

Meteora's decision for /var/lib/meteora

With all of the above, the choice can now be argued rather than copied from a tutorial. Meteora's workload profile:

Workload characteristic Value
File size 17.3 MB, one per day
Number of files ~365 a year, about 1,800 in total
Write pattern Append to the end, continuous, ~8 readings/second
Read pattern Full sequential (aggregator) and random via mmap (meteo-api)
Device Software RAID 1 over two NVMe drives (02-05)
Integrity requirement High: the data cannot be regenerated
Availability requirement High: meteo-api serves 24/7

The candidates and their reasoned rejection:

Candidate Argument in favor Why it is not chosen
XFS Excellent with large files, highly parallel, no inode limit It cannot be shrunk; the volume might need adjusting
Btrfs Data checksums, native snapshots Uneven performance with continuous writes; its RAID 5/6 is not reliable
ZFS The best at integrity Outside the kernel; complicates maintenance and upgrades
F2FS Designed for flash Aimed at phones; less mileage on servers
ext4 Mature, predictable, extents, tooling No data checksums

Decision: ext4, with this justification:

  1. Maturity and predictability. It is the file system with the most flight hours in the Linux ecosystem. For unrecoverable data, "boring and proven" is worth more than "modern and promising".
  2. Extents. A contiguous 17 MB file is described with a single extent instead of 4,219 pointers. We will measure it with filefrag in 04-05.
  3. Tooling. dumpe2fs, debugfs, tune2fs, e2fsck and resize2fs (which can shrink, unlike XFS) make up the most complete set. When something goes wrong at three in the morning, that counts.
  4. The journal in ordered mode guarantees that another file's data will never appear inside the readings after a failure. We will justify it in 04-05.
  5. Inodes are not an issue for this workload: 1,834 used out of 13 million.
  6. The lack of data checksums is compensated for by other means: md verifies the RAID 1 with a weekly scrub, and the ingestor stores a CRC32 per reading in its own format.

The exact formatting parameters:

sudo mkfs.ext4 \
    -b 4096 \                    # 4 KiB block = page size
    -i 1048576 \                 # one inode per MiB: few, large files
    -m 1 \                       # only 1 % reserved for root (default is 5 %)
    -L meteora-data \            # stable label
    -O extent,dir_index,has_journal,metadata_csum,64bit \
    /dev/md0

What each option does and why here:

  • -b 4096: it matches the page and the NVMe's physical sector; it also allows direct mmap().
  • -i 1048576: by default there would be 13 million inodes, of which 1,834 are used; with one inode per MiB there are 200,000 left, more than enough, and it saves about 3 GiB of inode table that becomes usable space. It is a safe adjustment only because we know the workload.
  • -m 1: ext4 reserves 5 % for root, so that a full disk does not prevent logging in or operating. On a 200 GiB data volume that means 10 GiB tied up; at 1 % it is 2 GiB, margin enough. Careful: on / never go below 5 %.
  • -O extent: extents instead of indirect pointers (04-05).
  • -O dir_index: hash indexes for large directories (04-02).
  • -O metadata_csum: metadata checksums, which detect corruption of the inode or the bitmap.

And that is how /var/lib/meteora/readings/2026-08-31.dat stops being a magic string and becomes inode 1180934 of the ext4 labeled meteora-data on /dev/md0.

Common Mistakes and Tips

Believing that ctime is the creation date. It is change time: it changes with chmod, chown or when a hard link is created. The real creation time is the crtime, which only stat shows on modern systems. Confusing them leads to false conclusions in any forensic analysis.

Misreading stat's Blocks field. They are always 512-byte blocks, even if the file system uses 4 KiB. If you multiply by 4,096 you will get a size eight times larger than the real one.

Not looking at df -i when faced with an ENOSPC. A disk at 26 % that reports "No space left on device" is almost always inode exhaustion. Always check both.

Assuming the number of inodes can be increased. In ext4 it is fixed at format time and there is no going back. If you foresee millions of small files, decide -i beforehand, or use XFS or Btrfs, which allocate them dynamically.

Choosing a large block "because it is faster". With a million 800-byte files, a 64 KiB block wastes 99 % of the space. The block size depends on the distribution of your files' sizes, not on an intuition about speed. And on Linux it cannot exceed the page's 4 KiB.

Running mkfs on the wrong partition. mkfs does not ask. Always confirm with lsblk and blkid before pressing Enter; we will see this in 04-03.

Tip: save the dumpe2fs -h output of your volumes. Having the UUID, the block size and the positions of the backup superblocks at hand turns an emergency recovery into a two-minute formality.

Tip: use stat instead of ls -l when something does not add up. It shows the four dates, the inode, the device and the real blocks. Half the mysteries involving files are solved by reading a stat calmly.

Tip: check the type before assuming. Before running cat on something unknown, look at the first character of ls -l. A cat on a 200 GiB block device, or on a FIFO with no writer, does not end the way you expect.

Exercises

Exercise 1: identifying types and reading a stat

Without using ls -l, write a command that classifies everything in /dev, /run and your home directory by file type, counting how many there are of each. Then take a regular file of at least 10 MB, run stat on it, and answer: (a) how many 4 KiB blocks does it actually occupy? (b) how much space is wasted in the last block? (c) do mtime and ctime match, and what does it mean if they do or do not?

Exercise 2: the block size trade-off

A 500 GiB volume is going to host one of two possible workloads. Workload A: Meteora's daily files, 17,280,000 bytes each, for 30 years. Workload B: 20 million 600-byte session files. For block sizes of 1, 4 and 64 KiB, calculate for each workload: total blocks needed, space actually occupied, absolute waste and percentage. Then state which size you would choose for each workload and whether a single volume can serve both.

Exercise 3: diagnosing a misleading ENOSPC

A colleague warns you: the session service on meteo-01 is failing with "No space left on device", but df -h shows the volume at 31 %. Write the complete diagnostic procedure — which commands you run, in what order and what you expect to see in each one — the technical explanation of why it happens, the immediate fix to restore the service and the definitive fix so it does not happen again. Include the formatting command you would use and justify its parameters.

Solutions

Solution 1

Classifying by type is done with find -type or, better, with stat's format:

for d in /dev /run "$HOME"; do
    echo "=== $d ==="
    find "$d" -maxdepth 1 -printf '%y\n' 2>/dev/null | sort | uniq -c | sort -rn
done

-printf '%y' prints a single letter with the type (f regular, d directory, l link, b, c, p, s), and sort | uniq -c counts them. Typical output:

=== /dev ===
    198 c        ← character devices: terminals, /dev/null, /dev/random
     42 b        ← block devices: disks and partitions
     28 d
     18 l
=== /run ===
     34 d
     12 s        ← service sockets, including /run/meteora/api.sock
      6 p        ← FIFOs, including /run/meteora/readings.fifo

For the questions, with a stat giving Size: 17280000 and Blocks: 33760:

(a) stat counts 512 B blocks: 33,760 × 512 = 17,285,120 bytes. In 4 KiB blocks that is 17,285,120 / 4,096 = 4,220 blocks. The data needs ⌈17,280,000 / 4,096⌉ = 4,219, so the extra block is extent tree metadata.

(b) 17,280,000 = 4,218 complete blocks + 3,072 bytes. The last block uses 3,072 out of 4,096, so 1,024 bytes are wasted: 0.006 % of the file. Irrelevant here, decisive with 800-byte files.

(c) If mtime and ctime match, the last operation was a content write (which updates both). If the ctime is later, some piece of metadata was modified after the write: a chmod, a chown or an ln. A ctime later than the mtime with no known cause is exactly what makes an analyst suspicious.

Solution 2

Workload A — one file of 17,280,000 B, 30 years × 365 = 10,950 files:

Block Blocks/file Occupied/file Waste/file Total waste
1 KiB 16,875 17,280,000 0 0
4 KiB 4,219 17,281,024 1,024 11.2 MB
64 KiB 264 17,301,504 21,504 235 MB

Over 189 GB of data, even 235 MB is 0.12 %. With 4 KiB, the waste is 0.006 %: negligible. What rules here is the number of blocks to manage, and 4,219 is far easier to handle than 16,875.

Workload B — 20,000,000 files of 600 B (12 GB of real data):

Block Occupied/file Total occupied Waste % wasted
1 KiB 1,024 20.5 GB 8.5 GB 41 %
4 KiB 4,096 81.9 GB 69.9 GB 85 %
64 KiB 65,536 1,310 GB 1,298 GB 99.1 %

With 64 KiB it does not fit: it would take 1.3 TB on a 500 GiB volume to store 12 GB of data.

Choice. Workload A: 4 KiB, for its affinity with the page and mmap(), with effectively zero waste. Workload B: 1 KiB, which saves 61 GB compared with 4 KiB. And an important warning for workload B: 20 million files need 20 million inodes, and the default mkfs would give 32.7 million — only just — so you have to set -i 2048 or use XFS.

A single volume for both? Technically yes, but it is a bad idea: the requirements are opposites (large block and few inodes versus small block and many inodes), and besides, mixing workloads means the millions of small files fragment the space the large ones need. Two separate volumes, which is exactly the partitioning argument we will see in 04-03.

Solution 3

Diagnostic procedure, in order:

df -h /var/spool/cache        # 1. space: 31 % → not this
df -i /var/spool/cache        # 2. inodes: 100 % → HERE IT IS
findmnt /var/spool/cache      # 3. confirm the device and the options
sudo find /var/spool/cache -xdev -type f -printf '%h\n' \
     | sort | uniq -c | sort -rn | head    # 4. who created them
sudo dumpe2fs -h /dev/sdb1 | grep -E 'Inode count|Free inodes|Inode size'

Step 2 is the diagnosis and step 4 identifies the culprit. -xdev matters: without it, find would cross into other file systems and count files that do not consume this volume's inodes.

Technical explanation. In ext4 the number of inodes is fixed and decided at format time, with one inode per 16 KiB by default. A 50 GiB volume therefore has 3,276,800 inodes. Every file, however small, consumes one. When they run out, creat() and open(O_CREAT) return ENOSPC — "No space left on device" — even though free blocks remain, because the kernel does not distinguish between the two resources in the error code. The 600-byte session files take one 4 KiB block and one inode each: the inodes run out once you reach 3.2 million, with only 13 GB used.

Immediate fix (restore the service):

sudo find /var/spool/cache/sessions -type f -mtime +7 -delete
df -i /var/spool/cache          # verify that there are free inodes now
sudo systemctl restart sessions

Deleting files older than 7 days frees inodes immediately. With millions of files it is better to use find ... -delete instead of rm -rf, which can overflow the command line.

Definitive fix, in three layers:

  1. Automatic purge: a systemd unit with a timer (module 7) or a tmpfiles.d entry that deletes expired sessions every hour.
  2. Reformat with the right density, after taking a backup:
sudo mkfs.ext4 -b 1024 -i 2048 -m 0 -L sessions /dev/sdb1

-b 1024 because the files are 600 bytes and with 4 KiB you waste 85 %; -i 2048 gives one inode per 2 KiB, that is 26 million inodes, far above the original 3.2; -m 0 because it is a temporary data volume where no root reserve is needed.

  1. Monitoring: alert when df -i goes above 80 %, exactly the way space is watched. Almost no monitoring system does it by default, and that is why this failure keeps showing up.

A reasonable alternative: mount /var/spool/cache as tmpfs, since sessions are volatile by nature. Both problems disappear at once, at the cost of RAM and of losing them on reboot (04-03).

Conclusion

A file is a named sequence of bytes, with a size and an owner, that the system places wherever it likes and the program reads as if it were continuous. That abstraction solves in one stroke the seven problems any application would face working on module 2's raw block vector, and it does so with the same strategy as virtual memory: a translation table that turns an orderly logical space into a scattered physical one.

A file has three parts in three places: the content in the data area, the metadata in the inode, and the name in the directory. The central consequence, from which half this module hangs, is that the name is not inside the file: inode 1180934 does not know it is called 2026-08-31.dat.

In UNIX there are seven types of file — regular, directory, symbolic link, block device, character device, FIFO and socket — and the last four take up an inode but zero data blocks: the FIFO /run/meteora/readings.fifo and the api.sock socket from module 3 are file system entries whose content lives in the kernel. ls -l tells them apart by their first character and stat reports everything: logical size, 512-byte blocks, device, inode, link count and the four dates.

Physically, an ext4 is divided into block groups of 128 MiB, each with a block bitmap, an inode bitmap, an inode table and a data area, preceded by the superblock — the identity card, with backup copies that save volumes — and the group descriptors. Going from an inode number to its position on disk costs two divisions and one multiplication, and that is the deep reason why the inode is a number.

The block size is a measurable trade-off: 4 KiB is the standard because it matches the memory page (and therefore the page cache and mmap()) and the modern physical sector; going lower makes sense with millions of tiny files, and going higher is not even possible on Linux. Inodes, in ext4, are fixed at format time: hence the ENOSPC with a half-empty disk that only df -i explains. And the four timestamps distinguish reading (atime), modifying content (mtime), modifying metadata (ctime) and creating (crtime); noatime removes 48,000 useless daily writes on meteo-01.

With the comparison of families in hand, Meteora chooses ext4 for /var/lib/meteora: maturity in the face of unrecoverable data, extents to describe 17 MB in a single run, the most complete toolset for three in the morning, and a deliberate tuning of -i 1048576 and -m 1 that recovers about 11 GiB for useful data.

We are left with the loose end we have been dropping in every section: the name. We know it is not in the inode, we know it lives "in the directory", and we have said that a directory is "a special file", but we have not opened one. What exactly is inside /var/lib/meteora/readings/? How does the system manage to turn the string /var/lib/meteora/readings/2026-08-31.dat into the number 1180934, and how many disk accesses does that cost? Why can a symbolic link end up broken and a hard one cannot? And what happens when a directory has a hundred thousand files inside it?

All of that is what the next lesson opens: Directory Structures.

Operating Systems Fundamentals

Module 1: Introduction to Operating Systems

Module 2: Resource Management

Module 3: Concurrency

Module 4: File Structures

Module 5: System Protection and Security

Module 6: Virtualization and Containers

Module 7: Administration and Troubleshooting in Practice

© Copyright 2026. All rights reserved