The Linux kernel has more than 30 million lines of code. MINIX 3's has about 12,000 in its privileged part. Both do essentially the same thing, but they have made opposite decisions about which code deserves to run with all the machine's privileges. That decision — how much to put inside the kernel and how much to leave outside — is the great architectural question of operating systems, and forty years on it is still not fully settled. In this lesson you will see the answers that have been given, understand why each one succeeds and fails at different things, and check on meteo-01 how Linux manages a pragmatic compromise with loadable modules. By the end you will be able to explain why a bug in a disk driver can bring down the entire server in one design and not in another.

1. The underlying question: what goes inside the kernel
2. Monolithic systems
3. Layered design
4. Microkernels and message passing
5. The performance cost of the microkernel
6. Hybrid kernels
7. Exokernels and unikernels
8. Loadable kernel modules in Linux
9. Comparison table
10. Applied example: a faulty driver on meteo-01

Remember from the first lesson: code running in kernel mode can do anything with the machine — access all of memory, program any device, disable interrupts. Code in user mode can only ask the kernel for favors.

From that comes the central tension of the design:

• The more code there is inside the kernel, the faster it runs. Two parts of the kernel communicate with a plain function call: a few nanoseconds, with no context switch and no data copying.
• The more code there is inside the kernel, the more fragile and insecure the system is. A stray pointer in any line of those millions can corrupt any structure in the system. There is no safety net: in kernel mode there is no memory protection like the one that does protect processes.

Every architecture we are going to look at is a different position on that balance between performance and isolation.

In a monolithic kernel, all the operating system's services — scheduler, memory manager, file systems, network stack, device drivers — are compiled into a single program that runs entirely in kernel mode, in one and the same address space.

How its parts communicate: through ordinary function calls. When the ext4 file system needs to read a block, it calls the disk driver's function directly. Cost: nanoseconds.

Advantages.

• Maximum performance. There are no boundaries to cross inside the kernel.
• Direct access to shared structures. The scheduler can look at a process's memory state directly without asking anyone.
• Conceptual simplicity for whoever programs inside it: everything is accessible.

Drawbacks.

• No internal isolation. A bug anywhere can corrupt everything.
• Hard to maintain. With millions of lines and everything reachable from everything, implicit dependencies proliferate.
• One part cannot be restarted without restarting the whole system.

Examples: Linux, the classic UNIXes, FreeBSD, MS-DOS (monolithic and, on top of that, with no protection at all).

A frequent clarification: Linux being monolithic does not mean it is badly structured. It has very clear internal interfaces (the VFS for file systems, the device model, the module API). What defines monolithic is not disorder but that everything shares the same privileged address space.

The layered design organizes the kernel into levels, where each layer can only use services from the layer immediately below it and offers services to the one above. The THE system (Dijkstra, 1968) was the canonical example, with six levels from 0 (scheduling) to 5 (the operator).

The advantage is verifiability: you can verify each layer assuming the lower ones are correct, which hugely reduces the reasoning effort.

The problem is that reality does not let itself be arranged into strict layers. Does the memory manager go below the file system? It seems so, because the file system needs buffers in memory. But the memory manager needs the swap area, which is on the disk, which the file system manages. There is a genuine circular dependency, and no layer ordering resolves it cleanly.

That is why pure layered design is barely used, although its influence is enormous: nearly every modern kernel is structured into approximate layers, with levels skipped when necessary.

The microkernel idea is to take isolation to the extreme: leave in kernel mode only what is impossible to do any other way, and move everything else out into ordinary user processes called servers.

What remains inside the microkernel is very little:

• Address space management (the bare minimum, because it requires touching the MMU).
• Basic thread scheduling.
• Inter-process communication (IPC), which is its star function.

Outside, as ordinary unprivileged processes: the file system, the network stack, the device drivers, the high-level memory manager, process management.

How the parts communicate now: through message passing. When an application wants to read a file, it sends a message to the file server; that server sends another message to the disk server; the reply travels back the other way. The microkernel does nothing but carry messages from one process to another.

What the diagram shows is the essential difference: in the monolithic one, the kernel-mode box is enormous and its parts call each other for free; in the microkernel, the privileged box is tiny and every interaction between components crosses the mode boundary twice.

The microkernel's advantages are real and notable:

• Fault isolation. If the disk driver fails, it is a user process that dies. The system can detect it and restart it without the machine going down.
• A verifiable kernel. seL4, a microkernel of about 10,000 lines, has a formal mathematical proof that its implementation meets its specification. This is simply impossible with 30 million lines.
• Minimal attack surface. A flaw in the file server does not give control of the machine, only of that server.
• Maintainability. Components have explicit, mandatory interfaces: no shortcuts are possible.

Examples: MINIX 3 (also used in the Intel Management Engine, which makes it one of the most widely deployed systems in the world without almost anyone knowing), QNX (very widely used in automotive and critical systems), L4 and seL4, GNU Hurd.

Here is the microkernel's historical problem, and it is worth quantifying because it is the reason it has not won.

Let's compare what it costs for meteo-api to read a block of the day's file:

On a monolithic kernel:

On a microkernel:

A context switch costs on the order of 1 to 5 µs, counting the flushing of caches and of the TLB. If we estimate 2 µs per context switch, the microkernel adds about 8 µs of overhead to an operation that takes 50 µs on an SSD: 16% more latency. And for operations resolved from the page cache (0.08 µs), the overhead would be 100 times the useful work.

This is exactly Linus Torvalds's argument in his famous 1992 debate with Andrew Tanenbaum, the author of MINIX. Tanenbaum maintained that the monolithic design was "a giant step back into the 1970s"; Torvalds replied that portability and performance mattered more than elegance. Both were right on their own ground, and history has proved them both right in different fields: Linux dominates servers, QNX dominates critical automotive systems.

An important and current nuance: the cost of message passing has come down enormously. Jochen Liedtke demonstrated with L4 in the 1990s that a very carefully crafted IPC implementation could be 10-20 times faster than Mach's, to the point of making the microkernel viable. Dismissing the microkernel by quoting 1990 figures is a frequent mistake.

A hybrid kernel adopts the microkernel's conceptual structure (components with clear interfaces, some services as servers) but runs most of them inside the kernel's address space to avoid the cost of message passing.

• Windows NT and all its descendants: it has a hardware abstraction layer (HAL), an executive with well-delimited subsystems and an internal microkernel, but drivers and the graphics manager run in kernel mode for performance. The decision to move the graphics subsystem into the kernel in NT 4.0 was exactly this trade-off: more speed in exchange for a graphics driver fault being able to bring down the system.
• macOS / XNU: the name stands for X is Not Unix. It combines the Mach microkernel (which provides IPC, memory management and threads) with a monolithic BSD server that provides the UNIX interface, file systems and networking, all in the same address space. It has the structure of a microkernel without paying the cost of message passing between its two halves.

The term "hybrid" has its detractors: Torvalds has pointed out that a microkernel whose servers all run in kernel mode is, functionally, a well-structured monolithic kernel. It is a fair criticism from the point of view of isolation, which is what really distinguishes the architectures.

Two more radical ideas worth knowing about even though their use is a minority one.

Exokernel (MIT, 1990s). It starts from a criticism: the operating system's abstractions, convenient as they are, impose decisions that sometimes hurt the application. A database knows better than the operating system how its own blocks should be cached, but it is forced to go through the kernel's generic cache.

The exokernel's proposal is that the kernel should only protect and multiplex the hardware, without abstracting it, and that each application should build (with a library) the abstractions that suit it. It was never adopted commercially, but its criticism is alive: current mechanisms such as O_DIRECT (bypassing the kernel cache), io_uring or user-space network access are exokernel concessions inside conventional systems.

Unikernel. It takes the idea to its practical extreme: the application is compiled together with the parts of the operating system it needs into a single image that boots directly on a hypervisor. There is no separation between application and kernel because there is only one application.

Applied to Meteora: a meteo-api unikernel would be an image of a few megabytes containing the HTTP server, the TCP/IP stack and a minimal read-only file system. It would boot in milliseconds and would include neither bash, nor ssh, nor a user manager, nor anything an attacker could exploit. Its drawbacks are equally clear: you cannot debug it by logging in over SSH, every change requires recompiling the entire image, and the tooling ecosystem is thin. Examples: MirageOS, IncludeOS, Unikraft.

Linux is monolithic, but not rigid. Loadable Kernel Modules (LKM) make it possible to add and remove kernel code on the fly, without rebooting. It is the pragmatic answer to monolithism's main practical drawback: not having to compile a different kernel for every hardware combination.

Very important: a module runs in kernel mode, with full privileges. There is no additional isolation whatsoever. Modules provide deployment flexibility, not robustness.

Column by column:

• Module: the module's name. ext4 is the file system, nvme the SSD driver, e1000e the Intel network card driver.
• Size: bytes of code it occupies in kernel memory. xfs takes 2.1 MB.
• Used by: how many other components depend on it. ext4 has 1 because there is a mounted file system using it. xfs has 0: it is loaded but unused, and could be unloaded to save memory and reduce attack surface.

To investigate a specific module:

What to look at here and why:

• filename: the path of the .ko (kernel object) file. It is code compiled for this exact version of the kernel; a module compiled for another version will not load.
• license: GPL v2: if a module does not declare a GPL-compatible license, the kernel is marked as tainted and many developers will not accept bug reports. It is a legal and technical pressure mechanism at the same time.
• depends: other required modules. Empty here; insmod would fail if dependencies were missing, whereas modprobe resolves them automatically.
• intree: Y: the module is part of the official kernel tree, it is not a third-party one. Out-of-tree modules (typically proprietary graphics card drivers) are a classic source of instability.

Module management:

• modprobe -r unloads. It only works if Used by is 0; if something is using it, it fails with Module xfs is in use. It is an elementary protection, because unloading a module in use would corrupt the system.
• modprobe (without -r) loads and resolves dependencies, unlike insmod, which loads a specific .ko file without resolving them.
• /proc/sys/kernel/tainted returns 0 if the kernel is clean. A different value indicates that something unofficial has been loaded or that a serious error occurred earlier. It is the first thing to look at when diagnosing unexplained instability.

A useful way to read this table: each architecture wins in the field where its priority is the one that matters. Linux dominates where performance and hardware variety rule. QNX dominates where a failure costs lives and the system has to be certified before a regulator. There is no better architecture in the abstract.

Let's imagine the network card's e1000e driver has a bug: under a rare condition — a malformed packet whose declared length is greater than the real one — it writes 64 bytes past the end of a buffer.

The driver runs in kernel mode, in the same address space as everything else. Those 64 bytes are written over whatever came next in kernel memory. Two scenarios:

• The visible scenario. The corrupted area contains pointers of a critical structure. When they are used, the kernel triggers a kernel panic: it stops completely. meteo-01 goes down entirely, along with ingestor, aggregator, meteo-api and everything else. Data not yet flushed to disk is lost (the 8 MB of Dirty we saw in the previous lesson) and the machine has to be rebooted.
• The worse scenario: silent corruption. The corrupted area contains page cache data, specifically part of the 2026-08-31.dat file. There is no visible failure. The system keeps working. For hours meteo-api serves readings with wrong values to Meteora's customers, and later the kernel flushes that corrupt cache to disk, making the damage permanent. Nobody finds out until a customer complains.

The second scenario is worse than the first, and it is the most serious consequence of the lack of isolation: it is not just that a bug can bring down the system, it is that it can not bring it down and corrupt data silently.

If the system does go down, it would leave a trail:

• dmesg -T shows the kernel message buffer with human-readable timestamps (-T).
• journalctl -k filters kernel messages only; -b -1 those from the previous boot, which is where the cause of the crash will be; -p err limits it to error level.

The network driver is a user process with its own address space. When it writes 64 bytes outside its buffer, one of two things happens:

• If the address is outside the pages assigned to the process, the hardware raises a page fault and the system kills the driver process. The rest of the system carries on intact.
• If the address falls within the driver's own memory, it corrupts its own data, but the corruption is confined: it cannot reach the page cache or the kernel's structures.

In MINIX 3 there is also a component called the reincarnation server that watches the drivers and restarts them automatically when it detects that they have died. The practical outcome would be:

• A network service interruption of a few milliseconds.
• Some station readings lost, which will be resent.
• ingestor, aggregator and meteo-api still alive, with their TCP connections possibly broken but their state intact.
• The data already received is not corrupted.
• An entry in the log and no phone call in the middle of the night.

With this example it seems obvious that the microkernel is better, and in robustness it is. But we have to be fair about the whole picture:

• That driver bug is very infrequent. The drivers in Linux's official tree are tested on millions of machines.
• The microkernel's performance cost is paid on each and every operation, not only when there are failures.
• Linux has its own mitigations: KASLR, read-only kernel memory, module signature verification, and projects such as Rust-for-Linux, which brings into the kernel a language that prevents by construction precisely this kind of overflow.

Which one to choose depends on the cost of a failure. For meteo-01, a crash means a few hours without weather data: annoying and expensive, but bearable, so Linux is the right choice. For a car's braking system, the cost of a failure is unacceptable, and that is why QNX is used there and the overhead is paid without argument.

• Believing that monolithic means badly structured. Linux is very well structured internally. Monolithic refers to the shared address space, not to the quality of the design.
• Thinking that loadable modules turn Linux into a microkernel. They do not: a module runs with full privileges and can corrupt the system exactly like code compiled inside. They provide flexibility, not isolation.
• Dismissing the microkernel by quoting 1990 performance figures. Liedtke's work with L4 improved the cost of IPC by an order of magnitude, and seL4 has shown that a formally verified microkernel is viable in production.
• Assuming that a kernel bug always shows up as a crash. Silent corruption is more dangerous precisely because it is invisible. Faced with inexplicably wrong data, checking /proc/sys/kernel/tainted and the kernel log should be a reflex.
• Installing out-of-tree modules without thinking. Third-party drivers are a statistically prominent cause of instability, and they taint the kernel, which complicates any later diagnosis.
• Tip: when you evaluate an architecture, do not ask "which one is better" but "how much does a failure cost here". That question settles the decision immediately.

For each situation, state which kernel architecture would be most suitable and justify your answer using the criteria of the cost of failure and the performance requirements:

1. A control system for a robotic operating theater.
2. A database server serving 50,000 queries per second.
3. A car's electronic control unit managing brakes and power steering.
4. A cloud microservice that is deployed thousands of times a day and only serves an API.

Examine the state of the modules on your own Linux machine (or on a virtual machine) and answer:

1. How many modules are loaded and how much kernel memory do they take up in total?
2. Is there any module with Used by at 0 that you could unload?
3. Is the kernel tainted? If it is, find out why.

Write down the commands you would use and explain what you are looking for with each one.

meteo-01 suffers unexplained reboots every two or three days, always in the small hours. There is no clear pattern in the load. Design a diagnostic plan of at least five steps aimed at determining whether the cause lies in the kernel (and in which part), justifying what information each step is after.

Solution 1

1. Robotic operating theater → microkernel (QNX, seL4 or similar). The cost of failure is a human life, so isolation and certifiability dominate every other criterion. In addition, this kind of system must pass certifications (IEC 62304, IEC 61508) that require demonstrating the software's behavior; with a 30-million-line kernel that is unachievable, whereas seL4 provides a complete formal verification. The performance required is modest: moving a robotic arm does not demand millions of operations per second, but rather that each one arrives on time.

2. Database server at 50,000 queries/s → monolithic (Linux). Here performance rules. At that rate, every microsecond of overhead per operation translates into real hardware costs. The overhead of message passing would be unacceptable. Besides, the cost of a failure is high but bounded: a crash means unavailability and recovery from the journal, not irreversible damage. The right answer to the risk here is not to change architecture but to replicate: several machines with failover. An additional note: databases are precisely the use case that motivated the exokernel criticism, and that is why they use O_DIRECT to manage their own cache.

3. Braking control unit → real-time microkernel (QNX is the de facto standard in the automotive industry). It combines the two toughest demands: an unacceptable cost of failure and strict deadlines. It needs isolation (so that a fault in the infotainment module does not touch the braking one) and determinism (a guaranteed response within a bounded time). It is the scenario where the microkernel's overhead is paid without argument, and where the separation of components also allows each one to be certified at a different criticality level.

4. Cloud microservice → unikernel, or alternatively a container on Linux. The unikernel fits well: the image is minimal (megabytes), it boots in milliseconds (important if it scales on demand), and its attack surface is tiny since it includes neither a shell nor utilities. Its main drawbacks — difficulty of debugging and the impossibility of logging into the machine — matter little in an immutable deployment where the response to a problem is to replace the instance. In practice, however, the dominant option today is a container on Linux, because the tooling ecosystem is incomparably better: a clear example of ecosystem maturity weighing as much as technical merit.

Solution 2

1. Number of modules and memory occupied:

• lsmod lists the modules; tail -n +2 discards the header line so that it is neither counted nor added up.
• wc -l counts the remaining lines, that is, the modules.
• In the second command, awk accumulates the second column (Size, in bytes) in the variable total and at the end (END) prints it converted to MB. A typical desktop system has between 80 and 150 modules and 20-40 MB; a minimalist server, considerably fewer.

2. Unloadable modules:

• $3 == 0 selects the lines whose third column (Used by) is zero, that is, modules nobody is using right now.
• An important warning: Used by at 0 does not guarantee that it is safe to unload. A file system module can be at 0 and be needed as soon as a device of that type is mounted; a sound module at 0 will stop working as soon as something is played. On a production server, modules are not unloaded lightly: the right approach is to blacklist them (/etc/modprobe.d/blacklist.conf) and verify after a controlled reboot.

3. Tainted kernel:

• If it returns 0, the kernel is clean and there is nothing more to investigate.
• If it returns another number, it is a bit mask where each bit indicates a reason. To interpret it:

The loop checks bit by bit which ones are set using a shift (1 << i) and a logical AND. The most common reasons are bit 0 (a proprietary module loaded, typically the NVIDIA driver), bit 12 (an out-of-tree module) and bit 7 (the machine previously suffered a serious fault it recovered from). This last one is the most relevant for diagnosing instability: it indicates that something bad has already happened even though the system is still standing. The second line looks for the explanatory message the kernel usually leaves at the moment it becomes tainted.

Solution 3

Step 1: was it a crash or an orderly reboot?

last -x shows the history of boots and shutdowns. If shutdown appears, somebody or something ordered it (an automatic update, a timer). If only reboot appears with no preceding shutdown, it was a crash. This distinction is the first fork in the diagnosis and saves investigating in the wrong direction.

Step 2: look for the trail in the previous boot.

-b -1 accesses the previous boot, which is where the cause will be. If the last messages show Kernel panic, Oops, BUG: or a Call Trace, we have the call stack of the failure and can identify the module involved. If the log stops abruptly with no error at all, that is highly suspicious of a hardware problem or a power cut, because a software failure almost always leaves something written.

Step 3: check for tainting and third-party modules.

A kernel tainted by an out-of-tree module is an immediate suspect. The loop goes through the loaded modules and flags the ones that are not part of the official tree.

Step 4: correlate with the small-hours activity.

The fact that the reboots are always in the small hours is the strongest clue in the statement. We have to find out what happens at that time: the 02:30 aggregator run, a backup, an automatic package update, an integrity scan. If the reboot consistently coincides with one of these tasks, that task subjects the system to a specific load (intensive I/O, memory pressure) that triggers the latent fault.

Step 5: rule out causes that are not the kernel's.

Before blaming the kernel, three usual suspects have to be ruled out: that the OOM killer acted (lack of memory, not a kernel fault), that the disk is degraded (the SMART attributes for reallocated or pending sectors give it away) and that there are RAM errors. A faulty RAM module produces exactly this picture: random crashes, with no software pattern, often under load. To confirm it, memtest86+ for several hours is the definitive test.

Step 6 (if nothing above is conclusive): enable persistent diagnostics.

kdump reserves memory to boot a secondary kernel after a panic and dump the crashed kernel's memory image to disk. It is the only way to analyze after the fact a panic that never made it into the log. In addition, if the machine does not respond at all, enabling the kernel watchdog allows an automatic reboot and at least bounds the downtime while the investigation goes on.

Kernel architecture comes down to one decision: how much code to run with all the machine's privileges. The monolithic design puts everything inside and gains performance at the cost of having no internal isolation; the microkernel leaves almost everything outside and gains robustness and verifiability at the cost of message passing; hybrids adopt the structure of the second with the performance of the first, sacrificing the isolation that gave it meaning. The layered design contributed a way of reasoning that survives even though its pure form is impractical, and exokernels and unikernels question, from the other extreme, whether the system's abstractions should be mandatory at all.

Linux solves the practical problem of monolithism with loadable modules, which provide deployment flexibility but, it bears repeating, no additional isolation. And the example of the faulty driver on meteo-01 leaves us with the key lesson: the lack of isolation can not only bring down the server, but something worse — corrupt data silently.

The general conclusion is that there is no better architecture, but there is a question that settles the choice: how much does a failure cost. For meteo-01, a few hours without data; for a braking system, a life.

All of this has revolved around a boundary we have used constantly without explaining it: the one separating user mode from kernel mode. In the next lesson, User Mode, Kernel Mode and System Calls, we will see exactly how that boundary works, how it is crossed step by step and how much crossing it costs.