Over the five previous lessons we have talked about an invisible boundary: the one separating your applications' code from the system's privileged code. It came up when discussing what the kernel is, when classifying architectures and when following an HTTP request. The time has come to look at it closely. In this lesson you are going to understand how the hardware makes it possible for the operating system to defend itself from the programs it runs, what happens exactly — instruction by instruction — when ingestor stores a reading on disk, and why an apparently trivial operation like write() costs hundreds of times more than an ordinary function call. By the end you will be able to read the output of strace, which is probably the most revealing diagnostic tool that exists on Linux, and you will understand why buffering is not an implementation detail but a first-order performance decision.

1. Why a program cannot touch the hardware
2. Dual mode: the mode bit and privileged instructions
3. Privilege rings
4. What a system call is
5. The journey step by step, from user to kernel and back
6. API, C library and system call: three different things
7. Categories of system calls
8. Example in C: write() versus fprintf()
9. Real observation with strace
10. The cost of a system call and why it pays to minimize them

Let's start with the problem. Suppose meteo-api, an ordinary program, could execute any CPU instruction and access any memory address. A single bug, or an attacker who had compromised the process, would be enough for it to:

• Read ingestor's memory and obtain other customers' data.
• Write directly to the disk, bypassing the file system and the permissions, and modify /etc/shadow.
• Disable interrupts and monopolize the CPU forever: not even the timer could take it back.
• Reprogram the page table to access all of physical RAM.

Notice a decisive nuance: it is not enough for the operating system to "not let it". If the program can execute the instruction that reprograms the MMU, it will execute it and the system will have no way of preventing it, because by then it has already run. The only possible solution is for the hardware to refuse.

And there lies the central idea of all protection in computing: an operating system's security does not rest on its code, but on a processor mechanism that software cannot circumvent.

The CPU has a mode bit in its status register indicating which mode it is running in:

Certain instructions are privileged: if an attempt is made to execute them in user mode, the CPU does not execute them and raises an exception. On x86-64, some examples:

You can verify in practice that the protection works:

What happened, line by line:

• __asm__ volatile ("cli") inserts the cli machine instruction directly into the program. volatile stops the compiler from reordering it or removing it as useless.
• The CPU, on encountering cli with the mode bit set to "user", does not execute it. It raises a general protection exception.
• The kernel handles that exception, checks which process it came from and sends it the SIGSEGV signal, which by default terminates the process.
• The second printf never runs.

This is exactly what we wanted to demonstrate: the refusal does not come from the operating system, it comes from the silicon. The operating system only decides what to do afterwards.

Here is the elegant part of the design. If a program could set the mode bit to "kernel", all the protection would be useless. That is why the bit cannot be modified directly. It only switches to kernel mode in three circumstances, and in all three control jumps simultaneously to an address the kernel fixed in advance:

1. A hardware interrupt (the disk finished, a network packet arrived). It is asynchronous, not caused by the program.
2. An exception (division by zero, page fault, privileged instruction). It is synchronous but involuntary.
3. A system call, that is, a voluntary request from the program.

In all three cases, the mode change and the jump are a single atomic hardware operation. There is no instant at which the CPU is in kernel mode running code chosen by the program. That atomicity is what makes the system secure.

The x86 architecture generalizes dual mode into four levels or rings, numbered from 0 (highest privilege) to 3 (lowest). The idea, inherited from Multics, was to allow intermediate levels: for example, drivers in ring 1, with fewer privileges than the kernel but more than applications.

In practice, almost every system uses only 0 and 3, for two reasons: because other architectures (ARM, RISC-V, MIPS) offer only two levels and using four would break portability, and because intermediate levels complicate the design considerably for a debatable benefit. A curious fact: rings 1 and 2 were indeed used in some virtualization techniques before hardware support existed.

Modern virtualization did add a level: the so-called ring -1 or VMX root mode, where the hypervisor runs, below the guest systems' ring 0. You will see it in Virtualization: Hypervisors and Virtual Machines.

A system call (or syscall) is the only legitimate door for a program in user mode to ask the kernel for something. It is a voluntary, controlled request to cross the boundary.

Three characteristics define it:

• The entry point is chosen by the kernel, not by the program. The program cannot jump to an arbitrary kernel address. It can only say "I want call number 1" and the kernel decides which code runs.
• The arguments are validated. The kernel checks that the pointers you pass it point to memory that really is yours, that the file descriptor exists, that the size is reasonable. No check is optional: each one plugs a security hole.
• The return goes back to user mode. When the call finishes, the mode bit goes back to "user" and execution continues right after the instruction that caused the entry.

Linux has about 350 system calls on x86-64. You can see the complete list:

That number on the left is the system call number, and it is what the program puts in a register to indicate what it wants. The numbers are stable forever within an architecture: if they changed, every existing binary would stop working. That is why read has been 0 and write 1 for decades.

Let's follow what happens when ingestor executes write(fd, &reading, 24).

Let's detail the phases:

1. Preparing the arguments (user mode). Linux's x86-64 calling convention fixes which register carries what:

Notice that r10 is used and not rcx for the fourth argument, unlike ordinary function calls. The reason is purely mechanical: the syscall instruction destroys rcx by saving the return address there.

2. The syscall instruction. A single machine instruction that, atomically:

• Saves the program counter (rip) in rcx and the flags (rflags) in r11.
• Loads into rip the address the kernel wrote at boot time into the special register MSR_LSTAR.
• Switches the mode bit to kernel.

None of this is controlled by the program: the destination address was fixed by the kernel long before.

3. Stack switch. The kernel cannot use the user process's stack, because its contents could be tampered with or point to invalid memory. Every process has a separate kernel stack (16 KB on Linux x86-64) and the kernel switches to it immediately. This is one of the details most often overlooked and it is essential for security.

4. Dispatch and validation. The kernel checks that rax is within the range of valid calls and consults the system call table to jump to the corresponding function (sys_write). Then it validates each argument. For the pointer, for example, it uses copy_from_user() instead of reading directly: that function checks that the address belongs to the process's space. A kernel that dereferenced a user pointer directly would have a critical vulnerability.

5. Doing the real work. sys_write locates the file from the descriptor, copies the 24 bytes into the page cache and updates the file's size.

6. Return. The result is placed in rax, the registers are restored and the sysret instruction returns the saved rip and flags, and sets the mode bit to "user".

7. Error translation. Here something happens that confuses many people. The kernel does not use errno: it returns the error as a small negative number in rax (for example, -13 for "permission denied"). It is the C library that, on seeing a value between −1 and −4095, does:

That is why errno is a C library variable, not a kernel one, and that is why it must be checked immediately after the failed call: any other library function may overwrite it.

This distinction is constantly confused and clearing it up saves a lot of later confusion.

The important relationships:

• A library function may make no system call at all. strlen(), malloc() when there is memory in its internal pool, or printf() when it only fills its buffer.
• A library function may make several. fopen() does open() and often fstat(). printf() with a full buffer does a write().
• A system call may have several different wrappers. open(), open64(), creat() and fopen() all end up in the same family of calls.
• You can invoke a system call without a library, with syscall(1, fd, buf, 24) or directly in assembly. It is almost never worth it.

A particularly interesting case in the opposite direction: some system calls do not cross into the kernel thanks to the vDSO (virtual Dynamic Shared Object), a small library that the kernel maps into every process's space. gettimeofday() and clock_gettime() read the time from a read-only shared page without switching mode. The reason is pure performance: they are such frequent calls that a special shortcut was made for them.

A pattern that repeats across all of them: system calls are deliberately few and low-level. There is no "read a configuration file" call, nor a "make an HTTP request" one. The kernel offers minimal primitives and the libraries build on top. Every system call is an attack surface and a promise of compatibility forever, so adding a new one is a decision taken with extreme care.

We are going to write the same Reading two ways and understand why they are not equivalent.

Analysis of version A (write):

• open(...) is a direct system call. It returns an int, the descriptor, or -1 on error.
• The if (fd == -1) check is not optional: almost every real failure of a production service comes from unchecked errors. strerror(errno) turns the numeric code into a readable message.
• write(fd, r, sizeof(*r)) hands the 24 bytes to the kernel immediately: one system call per reading.
• The written != sizeof(*r) check covers a case that surprises many people: write can write fewer bytes than requested and return a smaller number without that being an error. It happens above all with sockets and pipes. Ignoring it produces silent truncation.
• The result on disk is 24 binary bytes, unreadable with cat, but compact and of an exact size.

Analysis of version B (fprintf):

• fopen(...) is a library function that wraps open() and also allocates a buffer (normally 4096 bytes) and returns a FILE *, a C library structure, not a kernel one.
• fprintf(...) formats the data as text and copies it into the buffer in user memory. In this specific case, it makes no system call at all: the generated line takes about 35 bytes and fits with room to spare.
• fclose(f) flushes the buffer, which does cause a write(), and then closes the descriptor.
• The result on disk is 118;1756636800;27.4;61.0;1013.2, readable but of variable and larger size (35 bytes versus 24).

The table summarizing the difference:

The last row is the key one, and we will quantify it shortly. The second-to-last explains a very frequent phenomenon: a program that dies abruptly loses whatever it had in the C library's buffer, and that is why logs cut off right before the error you are looking for. That is what fflush() is for, and it is why stderr is unbuffered by default.

strace intercepts and displays every system call a process makes. It is the tool that turns everything above into something you can see.

Let's start with the command's arguments:

• -f also follows the process's threads and children. Without this, in a multi-threaded program you would only see a part.
• -T shows in <> the time each call took. It is the most useful option for diagnosis.
• -p 1099 attaches to an already running process, in this case ingestor. Alternatively, strace ./program launches it from the start.
• 2>&1 redirects the error output (where strace writes) to standard output, so it can be piped through head.

Now the interpretation of the cycle, which is the interesting part:

1. recvfrom(7, ...) = 48 — reads 48 bytes from the socket with descriptor 7: a station's packet. The = 48 is the return value. It took 9 microseconds.
2. clock_gettime(CLOCK_REALTIME, ...) = 0 — gets the time to stamp the reading. It took 1 microsecond: suspiciously little for a system call, and the reason is that it is resolved through the vDSO without crossing into the kernel.
3. write(9, ..., 24) = 24 — writes the Reading's 24 bytes to descriptor 9, the day's file. It took 21 µs.
4. recvfrom(...) = -1 EAGAIN — it tries to read from the socket again and there is nothing. EAGAIN on a non-blocking socket is not a real error: it means "there is no data right now, come back later".
5. epoll_wait(5, ..., 1000) = 1 — the process blocks waiting for activity, with a maximum time of 1000 ms. It took 31,472 µs, that is, 31 ms. During that time ingestor is in state S and consumes no CPU at all: the correct behavior of a service that waits.

To see the aggregate summary, which is usually more useful than the detail:

How to read this, which is where the tool's real value lies:

• epoll_wait takes 71% of the time, but that is good: it is time blocked waiting for work, not CPU consumed. A healthy service spends most of its time here.
• The errors column shows 26,000 errors in recvfrom. It looks alarming, but they are the expected EAGAINs from non-blocking polling. Many errors in strace -c do not mean there is a problem; you have to look at which ones they are.
• And here is the important finding: 52,000 calls to write for 26,000 readings received. That is exactly two writes per reading. Investigating the code we would discover that ingestor writes the reading to the data file and also a line to the log, with no buffering at all. That is the optimization target, and it takes us straight to the last section.

Let's put numbers on the cost of crossing the boundary.

Where does that cost come from? From several sources that add up:

• The mode and stack switch itself.
• Saving and restoring registers.
• Argument validation.
• And above all, since 2018, the Meltdown and Spectre mitigations. The main one, KPTI (Kernel Page Table Isolation), separates the kernel's and the user's page tables, which forces part of the TLB to be flushed on every transition. This mitigation multiplied the cost of system calls by a factor of between 2 and 5, and turned buffering from advisable into essential.

With 500 stations sending one reading per minute:

Current situation (two write calls per reading, no buffering):

With a 4 KB buffer for the data file. Since each Reading takes 24 bytes, 170 fit in 4 KB:

Reduction: a factor of 170. From 7.2 seconds to 0.04 seconds of CPU per day.

In absolute terms 7 seconds a day does not look like anything, and on meteo-01 with 500 stations it probably is not. But the reasoning changes completely if Meteora grows to 50,000 stations: we would go from 12 minutes of CPU wasted daily to 4 seconds, and with bursty load the difference shows up as latency, not as average consumption.

The buffer is not free. It trades performance for durability:

And there is a third level worth distinguishing precisely, because almost nobody is clear about it:

1. The C library's buffer (user space). It is flushed with fflush(). If the process dies, it is lost.
2. The kernel's page cache. The data is already outside the process: if the process dies, it survives. But if the machine powers off, it is lost.
3. The physical disk. It is forced with fsync(). Only here is the data truly durable.

The engineering decision for Meteora would be: use buffering for the reading data (losing 170 readings in a crash is acceptable, they can always be retransmitted) but not for critical alerts or for the audit log, where every entry must reach the disk whatever it costs. This same trade-off, in its most general form, will come back in Space Allocation, Journaling and Integrity.

• Believing that errno comes from the kernel. The kernel returns a negative value in rax; errno is a C library variable. Check it immediately after the failed call, before invoking any other function.
• Not checking write's return value. It can write fewer bytes than requested without that being an error. The correct pattern is a loop that retries with what is left.
• Confusing a library function with a system call. printf is not a system call; fopen is not open. strace is the definitive way to check what really crosses the boundary.
• Thinking that write() means "it is on the disk". It means it is in the kernel's page cache. Only fsync() guarantees durability, and only if the hardware is not lying about its own caches.
• Optimizing without measuring. Before redesigning anything, strace -c for a minute tells you exactly which calls dominate. In ingestor's case, the 52,000 writes showed up on their own.
• Overusing strace in production. It slows the observed process down considerably (it can multiply the cost of each call by 10 or more), because each one causes two stops of the process. For production, lower-impact tools such as perf or bpftrace are preferable.
• Tip: when a program is slow and you do not know why, run strace -c -f for thirty seconds. In most cases, the answer jumps out at you from the first line of the table.

For each of these functions in a C program, state how many system calls it causes approximately and why. Reason in terms of buffers:

1. strlen("2026-08-31.dat")
2. printf("Reading received\n") with the output redirected to a file.
3. printf("Reading received\n") with the output on an interactive terminal.
4. A loop that calls fprintf(f, "%.1f\n", temp) 1,000 times on a file opened with fopen.
5. A loop that calls write(fd, buf, 8) 1,000 times.

meteo-api responds to every HTTP request by writing a 120-byte line to /var/log/meteora/meteo-api.log with a direct write(), unbuffered. With 200 requests per second and a cost of 5 µs per system call:

1. How much CPU per day goes into those calls alone?
2. If an 8 KB buffer is added, how many calls would be left and how much CPU?
3. What is lost with that change and in which case should it not be made?

Write a C program that receives 1,000 simulated Reading structures and writes them to a file, minimizing the number of system calls without using the stdio library (that is, with a direct write()). Explain the design and compute how many calls it makes compared with the naive version.

Solution 1

1. strlen(...) → 0 system calls. It walks the process's own memory counting bytes up to the null terminator. It needs nothing from the kernel. It is the canonical example of a library function that does not cross the boundary.

2. printf redirected to a file → 0 calls on that invocation. When standard output is not a terminal, glibc configures it as fully buffered (4 KB). The 17 bytes are copied into the buffer and stay there. The write() call will happen when the buffer fills up (after about 240 messages), when fflush() is called or when the program terminates in an orderly way. An important practical consequence: if the program dies abruptly, the output file appears truncated, and frequently the message that explained the failure is exactly the one missing.

3. printf to a terminal → 1 system call. When the output is a terminal, glibc uses line buffering: it flushes the buffer on encountering a \n. Since the message ends in a newline, an immediate write() occurs. This explains a behavior that baffles many people: the same program shows its messages instantly on screen but seems to "write nothing" when redirected to a file. It is not a bug, it is a change of buffering policy.

4. 1,000 fprintf calls of ~6 bytes → approximately 2 calls. 6,000 bytes in total, with a 4,096-byte buffer: it is flushed once when it fills up and once on fclose. A reduction from 1,000 to 2.

5. 1,000 write calls of 8 bytes → exactly 1,000 calls. write() has no buffer: every invocation crosses into kernel mode. At 5 µs each that is 5 ms of CPU to write 8 KB, when with a buffer 2 calls and 10 µs would be enough. It is a difference of 500 times, and it is exactly the problem we detected in ingestor with strace -c.

Solution 2

1. Current situation:

A minute and a half of CPU a day devoted exclusively to crossing the boundary to write the log. Put another way: 0.1% of a core continuously, just for logging.

2. With an 8 KB buffer:

A reduction by a factor of 68, exactly the number of lines that fit in the buffer. From 86.4 to 1.27 seconds.

3. What is lost and when not to do it:

Two things are lost:

• Durability against a process crash: up to 68 log lines, corresponding to the last 0.34 seconds of activity. And here is the serious problem: they are precisely the lines describing what happened just before the failure, that is, the most valuable ones for diagnosing it.
• Immediacy of observation: an administrator running tail -f on the log will see the messages in jumps of 68, with variable delay. And monitoring tools that read the file will detect problems later.

It should not be done in three specific cases:

• An audit or security log. If it serves as evidence (who accessed what and when), losing the last entries is unacceptable. Besides, an attacker who caused a crash would erase the most incriminating trail along with it.
• An error log. It is best to apply the same policy as stderr: no buffering. The volume of errors is low by definition, so the cost is negligible and the diagnostic value is maximal.
• When there is an external traceability requirement (regulatory or contractual) demanding a record of every operation.

The correct engineering solution is to separate the streams: the access log, which is voluminous and not very critical, buffered; and the error and audit logs, which are sparse and critical, unbuffered. It is exactly what serious web servers do, and also the reason journald distinguishes priority levels.

Solution 3

Explanation of the design:

• CAPACITY = 170 is not arbitrary. 170 × 24 = 4,080 bytes, just under a page's 4,096. Aligning the buffer with the page size makes better use of the kernel's page cache and avoids a write unnecessarily straddling the boundary of two pages.
• flush_buffer() uses a loop, not a single write. This is what distinguishes correct code from code that works until one day it does not. write() can return fewer bytes than requested, and the loop advances the pointer p and decrements remaining until everything has been written.
• The handling of EINTR covers another real case: if a signal arrives while the process is blocked in write, the call can return -1 with errno == EINTR without having written anything. It is not an error: it has to be retried. Omitting this produces sporadic, irreproducible data loss, among the worst kinds to diagnose.
• append_reading() flushes before adding, not after. That way the buffer never overflows and the order of operations is always correct.
• The final flush_buffer() is essential. Without it the last 150 readings would be lost (1,000 − 5 × 170 = 150), because they would stay in the buffer unwritten. It is the exact equivalent of fflush(), and forgetting it is the most frequent mistake when implementing buffering by hand.

Comparison of system calls:

A reduction by a factor of 167. And notice a detail that reinforces what we saw in the lesson: the number of system calls does not depend on how much data you write, but on how many times you cross the boundary. Writing 4,080 bytes costs practically the same as writing 24, because the dominant overhead is the mode switch, not copying the data.

An optional addition for robustness: if these readings were critical, after flush_buffer() you would have to call fsync(b.fd) to force the flush to the physical disk. It costs on the order of milliseconds, so it is only justified when data loss in a power cut is unacceptable.

An operating system's protection does not rest on its code but on the hardware: the CPU's mode bit makes privileged instructions simply not execute in user mode, and the switch to kernel mode can only happen through an interrupt, an exception or a system call, always jumping to an address the kernel fixed in advance. Of x86's four privilege rings, in practice only 0 and 3 are used.

A system call is the only legitimate door through that boundary, and we have walked through its complete mechanics: the call number in rax, the arguments in registers, the syscall instruction, the switch to the kernel stack, argument validation, table-based dispatch, the return with sysret and the translation of the negative value into errno performed by the C library. You have also seen why API, C library and system call are three different things: printf is not a system call, fopen is not open, and strace is how you check what really crosses.

And above all you take away an idea with daily practical consequences: crossing the boundary costs, between 50 nanoseconds and several microseconds, and that cost does not depend on how much data you move. Hence buffering is not an ornament but a design decision that trades performance for durability, and in ingestor it would cut system calls by a factor of 170. Knowing where to put it — and where never to put it, as in an audit log — is one of those decisions that separate code that works from code that holds up in production.

With this you close module 1. You know what an operating system is, where it comes from, what types there are, what functions it performs, how its kernel is organized inside and how the boundary that protects it is crossed. From here on we stop looking at the system from the outside and start opening it up. In Module 2: Resource Management we will get into the first and most fundamental of its jobs: Process Management, where you will see what exactly lies inside that abstraction we have used in every lesson without ever opening it, how a process is born with fork and execve, what states it goes through and what happens in a context switch. Meteora's three processes will stop being names in a ps listing and become structures you know how to read.