We closed the previous lesson with three unopened boxes and a problem stated in numbers. The boxes: what exactly the CPU does when an interrupt arrives, how a driver registers with the kernel and what contract it fulfils, and how DMA works on the inside. The problem: at 500,000 readings per second, a whole core would be devoted solely to servicing interrupts, and that cannot be the final answer.

This lesson opens the three boxes and solves the problem. You are going to see the complete journey of an interrupt from the IRQ line to the handler, understand why a handler's code cannot sleep or block, and why that restriction forces the work to be split into two halves. You will learn to read /proc/interrupts line by line, which is one of the most informative outputs in the system. And you will follow the complete path of a packet carrying station readings, this time with no layer left unexplained.

It is the last lesson of module 2, so at the end we will recap all seven and link up with module 3.

1. What a device driver is
2. The contract with the kernel: the file operations interface
3. Driver registration and loadable modules
4. Kernel space versus user space for drivers
5. Interrupts: IRQ line, vector and vector table
6. The interrupt controller: from the PIC to the APIC
7. What the CPU does when an interrupt arrives
8. Interrupt context and its rules
9. Masking and shared interrupts
10. Top half and bottom half
11. /proc/interrupts interpreted line by line
12. MSI and MSI-X
13. DMA in detail: descriptors, coherence and the IOMMU
14. The complete path of a packet carrying readings
15. Interrupt latency and its impact
16. Closing module 2

A device driver is the code that translates the kernel's generic operations into the concrete operations of one hardware model.

It is worth separating two things that are constantly confused, and which in some languages even share a name:

In this lesson, when we say driver we mean the software; when we say device controller or chip, we mean the hardware.

The driver is the only piece of the system that knows the dirty details: which register has to be written to start a transfer, which bit carries the "ready" signal, how many microseconds you have to wait after a reset, what silicon erratum revision B2 of the chip has.

And their sheer size in the Linux kernel is revealing:

Drivers are 63 % of the kernel's code. And in the Linux source tree the proportion is similar: more than half of the millions of lines are device drivers. The kernel proper — scheduler, memory, file systems, network — is the small part.

A driver cannot do as it pleases: it has to fulfil a contract with the kernel. That contract is a structure of function pointers. For a character device:

How the mechanism works, which is the same table dispatch we saw in 01-06 with system calls:

• When a process calls read() on /dev/meteora/north-station, the kernel walks the path you already know: syscall → call table → sys_read → the VFS layer.
• The VFS layer looks at the file's major number, finds the registered driver and calls the .read pointer in its file_operations.
• That pointer points at meteora_read, code specific to this device.

It is polymorphism implemented with function pointers in C: the same read() call ends up running different code depending on the device, without the calling code knowing anything about it.

The usual methods in the contract:

ioctl deserves a comment. It is the escape hatch from the "everything is a file" model: it allows operations that are neither reading nor writing. Ejecting a CD, setting the speed of a serial port, querying a card's statistics. The fact that it exists demonstrates the limits of the file abstraction, exactly as sockets demonstrated the limits from another direction. ioctl is the pragmatic acknowledgement that not everything fits into read/write.

A simplified driver skeleton:

Points to understand about this code:

• copy_to_user() instead of memcpy(). It is mandatory and it is not a formality. The buf pointer comes from user space and is not to be trusted: it could point to kernel memory, to an unmapped address or to another process's memory. copy_to_user validates the range, handles the page fault if the page is not present, and returns the number of bytes it could not copy. A plain memcpy would be a textbook privilege escalation vulnerability. The __user annotation in the prototype also lets static analysis tools catch the mistake.
• Returning -EIO and -EFAULT, negative. That is exactly the convention we traced in 01-06: the kernel returns the error as a small negative value, and the C library turns it into -1 plus errno.
• register_chrdev with major 0 asks the kernel to assign a free number. Fixing a major by hand only works for the officially reserved numbers.
• module_init and module_exit define the module's entry and exit points, which is what makes it possible to load and unload it at run time.

Here we pick up what we saw in 01-05: loadable modules let you add code to a running kernel.

The alias line is the key to automatic loading: it says that this module manages the PCI device with vendor 8086 (Intel) and device 1521 (I350). When the PCI bus enumerates a card with those identifiers, udev looks for a module with the matching alias and loads it with no human intervention.

There is the complete chain: the hardware identifier leads to the module name.

Advantages of modules, picking up the kernel architecture discussion from 01-05:

And one important limitation: a module runs with all the kernel's privileges. There is no isolation. A buggy module can corrupt any structure in the system.

Here we pick up and settle the architecture debate from 01-05, now with concrete data.

Why a faulty driver is dangerous. A driver runs in ring 0, so:

• It can write to any physical memory address, including the kernel's structures and any process's memory.
• It can execute privileged instructions: disabling interrupts, changing the page table, reprogramming the MMU.
• There is no MMU to protect it from itself: the protection mechanisms we studied in 02-03 and 02-04 protect processes from each other, but the kernel sits above them.

A null pointer in a driver:

How to read this trace:

• supervisor read access in kernel mode: the fault happened in ring 0. Had it been in user space, it would be a simple SIGSEGV.
• RIP: meteora_interrupt_handler+0x2a [meteora_drv]: the exact address where it failed, with the guilty module named in brackets.
• Tainted: G O: the kernel is "tainted" by an out-of-tree module. Kernel developers use this mark to know that the problem may not be theirs.
• Fatal exception in interrupt: and here is the serious part. The fault happened inside an interrupt handler, where the kernel cannot simply kill the process responsible (there is no process: Comm: swapper/2 is the idle thread). No recovery is possible: kernel panic.

Compare it with the same bug in meteo-api: one process dies, systemd restarts it, a few requests are lost. Here the whole machine dies.

Drivers in user space. The alternative exists and is used in specific cases:

The DPDK case is paradoxical and very instructive: it moves the driver into user space not for safety, but for performance. It maps the card's registers straight into the process and polls continuously, eliminating interrupts and system calls. It manages to process millions of packets per second at the cost of dedicating whole cores to spinning in a loop. It is exactly the busy waiting of the previous lesson, chosen deliberately because at those rates it is cheaper than interrupting.

An interrupt is a hardware signal that makes the CPU suspend what it is doing and run a predetermined piece of code.

The vocabulary, which you need to be clear about:

On x86-64 there are 256 vectors, allocated like this:

Some exception vectors you already know from earlier lessons:

And here it is worth pinning down a classification that is often muddled:

The key difference is synchrony. An exception always happens at the same point if you rerun the program with the same data: it is a consequence of the instruction being executed. An interrupt arrives whenever the outside world feels like it and can land between any two instructions. That asynchrony is the source of all the difficulty in interrupt code.

The IDT lives in memory and its address is in the IDTR register, loaded with the privileged lidt instruction we saw in the table in 01-06. Each entry holds the handler's address, the segment and the permissions.

The CPU has very few interrupt pins, so an intermediary chip is needed to multiplex the lines from all the devices.

PIC 8259A (1976): two chained chips, 15 usable IRQ lines.

That 1981 assignment is still recognizable today in /proc/interrupts.

The APIC (Advanced Programmable Interrupt Controller) replaced it, and its improvements are what make a multiprocessor server viable:

The APIC has two parts:

• Local APIC: one per core, integrated into the CPU. It receives the interrupts directed at that core and manages the local timer and the inter-processor interrupts.
• I/O APIC: in the chipset. It receives the devices' lines and routes them to the Local APIC of the appropriate core.

The fact that the APIC can direct an interrupt at a particular core is what enables the smp_affinity we will see when we get to /proc/interrupts, and what makes it possible for a card with several queues to spread its work across cores.

This is the exact sequence, and it is worth comparing it mentally with the system call sequence from 01-06:

The details that matter:

1. The instruction in progress is finished. The CPU does not interrupt halfway through an instruction (with a few exceptions for very long instructions, which are restartable). This guarantees a consistent state.
2. The minimum state is saved automatically: instruction pointer, stack pointer, flags register and segment selector. It is done by the hardware, not by software.
3. It switches to the kernel stack, exactly as in a syscall. The handler never uses the interrupted process's stack.
4. Interrupts are disabled if the IDT entry is an interrupt gate (the usual case). This stops another interrupt from nesting immediately.
5. The EOI is mandatory. If the handler does not send the end-of-interrupt signal to the APIC, that device will never interrupt again. It is one of the classic mistakes when writing drivers, and its symptom is that the device works exactly once and then goes silent.

A comparison with what you already know:

The crucial row is the second to last: a system call runs on behalf of a particular process, whereas an interrupt runs in the context of whoever happened to be running, which could be anyone. Every restriction in the next section comes out of that.

An interrupt handler runs in interrupt context, also called atomic context. It is an environment with very strict rules:

Understanding the reason behind the first rule is the key to everything else. When the ingestor calls read() and there is no data, the kernel puts it in state S and schedules another process: there is a task_struct to park. But an interrupt handler does not belong to any process. If it went to sleep, who would be put in state S? The process that happened to be running, which has nothing to do with it? And who would wake it up?

Remember the kernel panic trace: Comm: swapper/2. The handler ran "on top of" the idle thread, which was what happened to be on that core. It was not its interrupt and not its work.

The four decisions in the correct handler:

1. Check whether the interrupt is its own before anything else, because the line may be shared. Returning IRQ_NONE lets the kernel try the next registered driver.
2. Acknowledge it in the hardware by writing to the status register. Without this the device would keep asserting the line and an endless interrupt storm would follow.
3. spin_lock_irqsave instead of mutex_lock. A mutex sleeps if it is taken; a spinlock spins waiting, which is the only thing you can do in atomic context. The irqsave variant also saves the interrupt state and disables interrupts, avoiding a deadlock against yourself if the same interrupt were to fire again.
4. Delegate all the real work to the bottom half. All it has done is bump a counter and schedule deferred work.

Masking means temporarily disabling an interrupt, and there are three levels:

cli is the privileged instruction we tried in 01-06 and which caused a SIGSEGV in user mode. Now you understand why it has to be so emphatically privileged: a process that could disable interrupts would stop the timer from taking the CPU away from it, monopolizing the CPU forever and completely breaking the preemptive scheduling of 02-02.

NMIs (Non-Maskable Interrupts) cannot be masked under any circumstances. They are reserved for catastrophic failures: a memory parity error, the watchdog timer, hardware failure signals. If the system stops responding entirely, the watchdog NMI is what forces a diagnostic dump.

Shared interrupts. With few IRQ lines and many devices, several can share a line:

IRQ 16 is shared by the USB controller and the SMBus. When it fires, the kernel does not know which of the two it was, so it calls the handlers of every driver registered on that line, in order, until one returns IRQ_HANDLED.

Hence the importance of the check in the previous section: a driver that returned IRQ_HANDLED without checking whether the interrupt was its own would steal the other device's interrupts, and that device would stop working with a symptom that is impossible to diagnose.

The last argument is essential on shared lines: it is what lets the handler know which device instance the call belongs to.

Here is the central idea in Linux's interrupt design, and it resolves a real tension.

The tension: an interrupt handler has to be extremely fast, because it runs with interrupts disabled and blocks everything else. But the work an interrupt generates can be considerable: processing a network packet means walking the IP stack, finding the socket, updating statistics, waking processes.

The solution: split it in two.

The three bottom-half mechanisms in Linux:

• softirq: the fastest and the only one that scales. There is a fixed number defined at compile time, so only the kernel's main subsystems use it. The same softirq can run simultaneously on several CPUs, which requires its code to be reentrant.
• tasklet: built on top of softirqs, easier to use. A given tasklet never runs on two CPUs at once, which simplifies synchronization at the cost of scalability.
• workqueue: runs in a kernel thread, that is, in process context. It is the only one that can sleep, so it is the option when you need to allocate memory with GFP_KERNEL, wait on a mutex or do I/O.

Reading this output:

• NET_RX is 30 times higher on CPU0 (1,284,102 against ~40,000). All the incoming network traffic is processed on a single core. With a single-queue card that is what you would expect; with a multi-queue one, it means the interrupt affinity is not spread properly.
• TIMER is spread evenly, as it should be: every core has its own APIC local timer.
• BLOCK are the completed block device interrupts, correlated with the RAID activity we saw in 02-05.

The split is clear: the top half takes less than 1 µs and only schedules work. The bottom half processes up to 64 packets walking the whole network stack, and can take tens of microseconds, but with interrupts enabled, so it blocks nothing.

Here we solve the problem we left open. The key is in igb_ring_irq_enable: while there are packets to process, that queue's interrupts are disabled and the kernel polls actively.

NAPI is an adaptive hybrid: interrupts at low load (minimum latency) and polling at high load (maximum throughput). It switches mode on its own, with no configuration.

The numbers for Meteora:

It is one of the most elegant solutions in the Linux kernel: recognizing that neither of the two techniques from the previous lesson is always better, and switching between them according to the load.

The structure of a line:

A line-by-line analysis of what this system is saying:

The diagnosis that jumps out: enp3s0-rx-0 concentrates 1.28 million interrupts on CPU0 while the other three cores are at zero. At 800 readings/s it is not a problem, but it is the bottleneck that would show up as things grew.

The mask 1 (binary 0001) means "CPU0 only". To spread it out:

And there an interesting decision appears that links back to 02-02: in the scheduling lesson we confined the aggregator to cores 2 and 3 with taskset. Directing the network interrupts to CPU0 and CPU1 completes that partition: packet processing and the computation of averages stop competing both for CPU and for cache.

The irqbalance daemon does this spreading automatically, but on systems with strict latency requirements it is usually disabled so that affinity can be pinned by hand.

Watching the activity in real time:

It is the quickest way to check whether a device is generating interrupts at all. If a device is not responding and its counter is not increasing, the problem is in the hardware or in the interrupt routing, not in the software above.

Classic interrupts use physical lines: a dedicated pin from the device to the APIC. That has three problems:

1. Scarcity: there are few lines, hence the shared ones.
2. Race conditions: the device can assert the line before the data it has written by DMA has reached memory. The handler would read incomplete data.
3. One vector per device: you cannot tell "I have received a packet" from "I have finished transmitting".

MSI (Message Signaled Interrupts) does away with the lines: the device writes a value to a special memory address, and that write is the interrupt.

The two decisive advantages:

The race with DMA disappears. Because the interrupt is a write over the same PCIe bus as the data, the bus ordering guarantees that the data has already arrived when the interrupt does. The physical line took a different path and could overtake it.

MSI-X allows one vector per queue and per core. That is what makes possible the perfect spread of the NVMe device we saw above:

Remember from 02-05 that NVMe has up to 65,535 queues and that its advantage is parallelism. MSI-X is the piece that completes it: each queue has its own interrupt vector directed at its own core, so there is neither a shared lock nor a CPU centralizing the work.

Enable+ confirms it is active, and Count=5 that the card has 5 vectors: control, two receive queues and two transmit queues.

DMA transfers data between the device and memory without the CPU. Here is how it works on the inside.

Modern cards do not receive one command per transfer: they work with descriptor rings, structures in memory that the CPU fills in and the device consumes.

The life cycle:

This ring design is what makes reception at full speed possible: the card can write several packets with no CPU involvement at all, and the driver collects them in a batch when its turn comes. It is the same idea as the double buffer in 02-06, generalized to 256 slots.

Here there is a subtle and very real problem. The CPU has caches; DMA writes straight into RAM, bypassing them.

The solutions, in order of preference:

On x86 the hardware is coherent and the problem does not arise. On ARM and other architectures you have to synchronize explicitly, which is why Linux's DMA API is portable:

Notice the concept these calls express: ownership of the buffer is handed over between the CPU and the device. While it belongs to the device, the CPU must not touch it, and vice versa. It is a pattern that will reappear under another name in module 3.

A first-order security problem: DMA uses physical addresses and bypasses the MMU. A compromised card — or one with malicious firmware, or a Thunderbolt device plugged in by an attacker — could write to any physical address, the kernel's memory included. That is the DMA attack, and it has been exploited in practice.

The IOMMU (Intel VT-d, AMD-Vi) is an MMU for devices:

It is exactly the same mechanism as the MMU in 02-04 — translation tables, permission checks, a fault if it does not add up — but applied to devices instead of processes. Each device has its own DMA address space and can only reach what has been assigned to it.

Besides security, the IOMMU enables device passthrough to virtual machines: you can hand a physical card to a VM with the guarantee that its driver, even if compromised, cannot touch the host's memory or that of the other VMs. It is an essential piece of virtualization, and it will come back in Virtualization: Hypervisors and Virtual Machines.

Now, at last, the complete journey with no black boxes. A station sends a 24-byte reading over UDP and the ingestor receives it.

Step by step, with the cost and the reference lesson:

Two observations that sum up the whole module:

The CPU spends 12 µs on a packet that took 0.5 µs to arrive. The cost is not in moving 66 bytes, but in crossing layers: the interrupt, the interrupt context switch, the network stack, waking the process, the system call. It is the same lesson as the cost of the syscall in 01-06 and of the context switch in 02-01, applied to I/O.

Step 11 is the one that varies most. Steps 1 to 10 are deterministic and add up to ~12 µs. Step 11 depends on the load and on the ingestor's nice value, and can be zero or several milliseconds. The real latency is dominated by scheduling, not by the hardware, which is why in 02-02 we gave the ingestor a raised priority.

Interrupt latency is the time from the device raising the interrupt to the handler starting to run.

Its components:

The critical component is the third: if some other kernel code has interrupts disabled, the interrupt waits. That is the worst case, and it determines how predictable the system is.

An average latency of 4 µs and a maximum of 87 µs. For Meteora that is excellent. For industrial control with 50 µs deadlines, that 87 µs maximum would be a missed deadline.

PREEMPT_RT achieves that guarantee by turning almost all interrupt handlers into schedulable kernel threads and replacing spinlocks with preemptible mutexes. It reduces peak throughput in exchange for predictability, which is exactly the real-time trade-off we set out in 01-03: predictable, not fast. The detail belongs to Mobile and Real-Time Operating Systems.

Diagnostic tools:

rx_missed_errors and rx_no_buffer_count are the critical metrics for Meteora. If they are non-zero, readings are being lost: the card received packets but there were no free descriptors to write them into, because the driver did not recycle them in time. Since lost readings are unrecoverable, these two figures should be monitored with an alert.

Going from 256 to 2,048 descriptors gives the system eight times more headroom to absorb bursts before dropping packets. The cost is memory (2,048 buffers of 2 KB = 4 MB) and slightly more latency in the worst case. For a system where losing data is irreversible, it is a clearly favorable trade.

Doing heavy work in the top half. It is the number one design mistake in drivers. With interrupts disabled, the entire system waits. A handler that takes 500 µs makes the network card drop packets and causes xruns in audio. The rule: acknowledge the hardware, save the bare minimum, schedule the bottom half, get out.

Using mutex_lock in interrupt context. A mutex sleeps, and in atomic context sleeping hangs the system. Use spin_lock_irqsave. And do not forget the irqsave: without it, if the same interrupt fires again while you hold the lock, you deadlock against yourself.

Forgetting the EOI or the hardware acknowledgement. The symptom is unmistakable: the device works exactly once and then goes silent, or it generates an endless interrupt storm that freezes the machine.

Returning IRQ_HANDLED without checking whether the interrupt is yours. On a shared line you steal the other device's interrupts, and it stops working with a symptom impossible to connect to your driver.

Using memcpy instead of copy_to_user. It is a privilege escalation vulnerability, not a matter of style. The pointer comes from user space and is not to be trusted.

Ignoring rx_missed_errors. It is the metric that tells you whether you are losing data at the card. At Meteora, where lost readings are unrecoverable, it should have an alert configured.

Disabling irqbalance without pinning affinity by hand. You end up with every interrupt on CPU0, which is the worst of both worlds: no automatic spreading and no manual spreading.

Diagnostic tip: faced with a device that does not respond, the order is: watch -n1 cat /proc/interrupts (is it generating interrupts?), dmesg -T | tail -50 (are there driver errors?), ethtool -S or the equivalent tool (are there losses?), cat /proc/irq/N/smp_affinity (are they all on one core?). If the interrupt counter does not increase, the problem is below the driver: interrupt routing, misconfigured MSI or hardware. If it increases but no data arrives, the problem is above.

This is the state of meteo-01 during the 8:00 peak:

And in parallel:

1. Identify the main problem and justify it with at least three pieces of data from the two outputs.
2. Why is LOC 3.6 times higher on CPU0 than on the others?
3. What does it tell you that nvme0q1 is spread out and enp3s0-rx-0 is not?
4. Propose a concrete solution with the exact commands, and explain what each one would improve.
5. How would you check whether readings are being lost, and what would you do if they were?

You are writing the driver for a device that collects data from a group of stations connected over a proprietary bus. When a batch of readings arrives, the driver has to:

• (a) Read the device's status register. Takes 0.5 µs.
• (b) Acknowledge the interrupt by writing to a register. Takes 0.3 µs.
• (c) Copy 64 readings of 24 bytes from the DMA buffer. Takes 8 µs.
• (d) Validate the checksums of the 64 readings. Takes 45 µs.
• (e) Allocate memory to store them. It may need to sleep.
• (f) Write them to a file in /var/lib/meteora/readings/. Takes milliseconds.
• (g) Wake the process waiting in read(). Takes 0.3 µs.

Answer:

1. Which operations go in the top half and which in the bottom half? Justify each one.
2. Which bottom-half mechanism would you use for each group: softirq, tasklet or workqueue? Why?
3. Calculate the time spent with interrupts disabled in your design and compare it with doing everything in the top half.
4. If the device generates 2,000 interrupts per second, calculate the CPU percentage in both designs.
5. What would happen if you put (e) or (f) in the top half?

The Meteora team reports that readings are missing: the day's data has gaps. You investigate and find:

1. Locate where the packets are being lost. There are at least two distinct loss points: identify them and explain how they differ.
2. Calculate the loss percentage and what it means in readings lost per day.
3. Propose a complete solution for each loss point, with concrete commands.
4. There is an obvious configuration mistake in the ps output. Identify it and correct it, relating it to what we saw in 02-02.
5. Design the monitoring that would have caught this before the team noticed.

Solution 1

1. The main problem: CPU0 saturated by network processing.

The evidence:

The picture is unambiguous: CPU0 is 92 % busy while the other three are less than half loaded. The system has capacity to spare in aggregate (49.2 % global %idle) but one core is the bottleneck.

And the figure that makes it serious: if CPU0 saturates completely, the NET_RX softirq will not be able to drain the descriptor ring in time and readings will start to be lost, which at Meteora is irreversible.

2. Why LOC is 3.6 times higher on CPU0.

LOC is the APIC local timer interrupts, which drive scheduling. On a system with NO_HZ (tickless), the timer stops on idle cores to save power and only fires when there is work.

CPU0 has 7.7 % %idle: it is working almost all the time, so its timer practically never stops. The other three, at 53-68 % idle, spend long periods without ticks.

A high LOC is not the cause of the problem: it is one more symptom that CPU0 never rests.

3. nvme0q1 spread out against enp3s0-rx-0 concentrated.

The difference lies in the queue architecture, and it is exactly what we saw in 02-05 and in the MSI-X section:

NVMe: the standard defines one submission and completion queue per core. Each queue has its MSI-X vector directed at its core. When a process on CPU2 does I/O, it uses CPU2's queue and its interrupt lands on CPU2. No shared locks and no concentration.

The near-perfect spread confirms it is working as it should.

The I210 card: it only has one receive queue (rx-0, confirmed by the MSI-X Count=5: control, 2 RX and 2 TX at best, and here only one is used). All the incoming traffic goes through it, and its vector is directed at CPU0.

It is a combined hardware and configuration limitation, which is why modern server cards have multiple RX queues with RSS (Receive Side Scaling), which spreads the flows across queues by hashing the headers.

4. A concrete solution.

a) Check how many queues the card supports:

If it supports 4, enable them:

What it improves: the card now has 4 receive queues, each with its own MSI-X vector. With RSS, packets from different flows are spread across the four by hash, and each queue interrupts a different core. The %soft work is divided by four.

b) Spread the interrupt affinity:

What it improves: it guarantees the spread instead of leaving it to chance.

c) If the card supports only one queue, enable RPS in software:

e is 1110 in binary: CPU1, CPU2 and CPU3. What it improves: RPS (Receive Packet Steering) does in software what RSS does in hardware: the top half still runs on CPU0, but the network stack processing is distributed to the other three cores. CPU0 is deliberately excluded to unload it.

d) Tune the interrupt moderation:

What it improves: the card waits up to 50 µs before interrupting, grouping several packets per interrupt. With 8.4 million interrupts, grouping them ten at a time brings them down to 840,000. The cost is up to 50 µs more latency per packet, perfectly acceptable at Meteora, where not losing readings matters far more than delivering them 50 µs sooner.

e) Coordinate with the process affinity from 02-02:

What it improves: having the ingestor run on the same core where its packets are processed exploits cache locality: the packet's data is already in that core's L1/L2. It is the processor affinity of 02-02 applied to I/O.

5. Checking for lost readings.

How to read each loss point:

If there were losses, the actions in order:

Solution 2

1. Splitting between halves.

A discussion of (c), which is the interesting decision: the 8 µs copy could go in either half. The arguments:

• In favor of the top half: if the DMA buffer is a small ring, it has to be emptied soon so that the device can carry on writing. Delaying it risks data loss.
• In favor of the bottom half: 8 µs is 16 times the rest of the top half, and with interrupts disabled that is a lot.

The professional answer is not to copy at all. The correct design uses a descriptor ring like the network card's: the top half only notes which descriptors are ready (advancing an index, ~0.2 µs) and the bottom half processes the data directly in the DMA buffer, without copying it. That is exactly what igb_clean_rx_irq does.

The final design:

2. The mechanism for each group.

For (d), validating checksums: a tasklet.

• It does not need to sleep: it is pure computation over data already in memory.
• It has to run soon: the data is occupying the DMA buffer.
• A softirq is not needed: softirqs are a scarce resource, with a fixed number compiled into the kernel, reserved for the main subsystems (network, block, timers). A specific driver uses a tasklet, which is built on softirqs and is the interface designed for drivers.
• An extra advantage: a given tasklet never runs on two CPUs at once, which enormously simplifies synchronization.

For (e), (f) and (g): a workqueue, necessarily.

• (e) can sleep. kmalloc(GFP_KERNEL) blocks if there is no immediately free memory, waiting for the kernel to reclaim pages (the whole mechanism of 02-04). In atomic context that hangs the system. The workqueue is the only one of the three that can sleep, because it runs in a kernel thread, that is, in process context with its own task_struct.
• (f) does disk I/O. Milliseconds of blocking. Unthinkable outside process context.
• (g) has to come afterwards, so it goes with the previous two.

3. Time with interrupts disabled.

The two-half design:

Everything in the top half:

4. CPU percentage at 2,000 interrupts/s.

The two-half design:

(The workqueue does I/O: its time is mostly waiting on the disk, not CPU.)

Everything in the top half:

The metric that matters is the second column. Total consumption hardly changes: the work has to be done either way. What changes radically is how long the rest of the system is blind.

With interrupts disabled 10.8 % of the time, the impact on the rest of meteo-01 would be:

5. If (e) or (f) went into the top half.

With (e), kmalloc(GFP_KERNEL):

If there is immediately free memory, kmalloc returns without sleeping and it appears to work. When the system is under memory pressure — precisely when it matters most — it will try to sleep in atomic context and the system will hang or panic. It is the worst kind of bug: it works in testing and fails in production under load.

The correct alternative if you really had to allocate in atomic context is GFP_ATOMIC, which never sleeps but can fail more often and draws on an emergency reserve. Even so, here the design solution (a workqueue) is better.

With (f), writing to a file:

Worse still. Writing involves the file system, the page cache, the I/O scheduler from 02-05 and waiting on the disk: milliseconds of blocking, with multiple points where the code sleeps.

The chain of consequences:

The general conclusion of the exercise: the split into two halves does not exist to share out work, but to minimize the window in which the system cannot react to the outside world. Anything that does not absolutely have to be done right now must be moved out of the top half.

Solution 3

1. The two loss points.

Point 1: at the network card (hardware).

The card received the packets correctly off the wire but had nowhere to put them: the 256-descriptor ring was full because the NET_RX softirq had not drained it in time. The packets stayed in the card's internal FIFO until it overflowed.

Those packets never reached RAM. The kernel never saw them.

Point 2: in the kernel's backlog queue.

Column 2 is zero, so nothing was dropped in the backlog. But column 3 is the revealing one: 167,745 times the NET_RX softirq exhausted its time or packet budget and had to yield before draining the ring.

And there is the causal connection between the two points: a high time_squeeze is the cause of rx_missed_errors. The softirq cannot keep up, the ring is not drained, and the card drops packets.

Besides that, the second line of softnet_stat (CPU1) shows 18,962 packets processed against 16.9 million on CPU0: all the processing is on one core, the same problem as in exercise 1.

The difference between the two points:

The socket queue (ss) shows Recv-Q: 0, so the ingestor is reading in time: there is no third loss point. The problem is upstream of the application.

2. The loss percentage.

In daily readings:

285,811 readings lost per day, unrecoverable. Put in context:

0.41 % sounds small, but for hourly averages per station it means systematic gaps in the series, and in the worst case a particular station could lose entire runs if its traffic coincides with the moments of overflow.

3. A solution for each loss point.

For the loss at the card:

Why each one works:

• (a) Gives eight times more headroom to absorb bursts. With 2,048 descriptors at 800 packets/s, the system has 2.5 seconds of cushion instead of 0.3. It costs 4 MB of RAM.
• (b) Spreads the work across four cores, attacking the root cause.
• (c) Reduces interrupts by batching packets, leaving more CPU for the real processing.

For the time_squeeze:

• (d) netdev_budget is how many packets the softirq processes before yielding. Raising it from 300 to 600 halves the time_squeeze count. The risk is that the softirq hogs more CPU, so it is best raised in moderation.
• (f) Distributes the network stack processing to CPU1-3 even though the interrupt still lands on CPU0.

Making it persistent:

4. The configuration mistake in ps.

The priorities are inverted. The aggregator has a raised priority (−5) and the ingestor a normal one (0). It is exactly the opposite of what it should be, and it contradicts what we established in 02-02:

And this directly aggravates the loss problem. The full causal chain:

1. The aggregator at nice −5 gets ~75 % of the CPU under contention (the weight table in 02-02).
2. It is a CPU-bound process: when it runs, it holds the CPU for whole milliseconds.
3. The NET_RX softirqs compete with it for CPU0.
4. Delaying the softirq means not draining the ring → time_squeeze → rx_missed_errors.
5. The badly prioritized aggregator is causing readings to be lost.

The correction:

Made persistent in systemd:

This ties the module's three pieces together: CPU priority (02-02), processor affinity (02-02) and interrupt affinity (02-07), coordinated so that the path of the readings — card → softirq → socket → ingestor — does not compete with the computation of averages at any point.

5. Monitoring that would have caught this earlier.

The process failure here is as serious as the technical one: the problem was spotted by the data team seeing gaps, not by the system. By the time somebody looks at the series, unrecoverable readings have been lost for days.

Metrics and thresholds:

The key criterion: for rx_missed_errors the warning threshold is any non-zero increment, because each unit is a reading lost forever. It is not a performance metric, it is a data-loss metric.

Collection script:

The end-to-end check, the most valuable of all:

This last one is what really matters: it measures the outcome, not the intermediate indicators. There may be losses at a point you are not watching — the switch, the cable, the station itself — and only the end-to-end check catches them all.

They compare like this:

A decent monitoring system has all three: the cause metrics give you time to act, the symptom metrics pinpoint the exact place, and the outcome metric guarantees that nothing slips through. We will pick this approach up again in Performance Monitoring and Troubleshooting.

A device driver is the software that translates the kernel's generic operations into the concrete operations of one hardware model, and drivers are 63 % of the Linux kernel's code. They fulfil a contract — a structure of function pointers such as file_operations — that implements polymorphism in C: the same read() ends up in different code depending on the device's major number. They are loaded as modules whose alias connects the hardware's PCI identifier with the module's name, which is what makes automatic loading work. And they live in ring 0 with no safety net: the Fatal exception in interrupt trace we read does not end in a SIGSEGV, it ends in a kernel panic.

An interrupt arrives over an IRQ line, is translated into a vector, and the CPU jumps to the corresponding entry in the IDT, saving state and switching to the kernel stack just as in a system call. The decisive difference is that an interrupt belongs to no process: it runs on top of whoever happened to be there, which forbids sleeping, blocking, allocating memory with GFP_KERNEL or touching user space. Out of that prohibition comes the split into a top half — microseconds, acknowledge the hardware and schedule work — and a bottom half — softirq, tasklet or workqueue, with interrupts enabled. In the exercise we measured the difference: 1.1 µs against 54.1 µs of system blindness, a factor of 49.

NAPI solves the problem we left open in the previous lesson: it alternates between interrupts at low load and polling at high load, cutting the cost 65-fold at 500,000 packets/s. MSI-X does away with physical lines, with races against DMA and with the one-vector-per-device limit, allowing one interrupt queue per core — which is what makes nvme0q1 perfectly spread and enp3s0-rx-0 not. DMA works with descriptor rings that let the card write several packets without the CPU, requires cache coherence to be coordinated, and needs the IOMMU so that a compromised device cannot write to any physical address it likes. And /proc/interrupts tells this whole story line by line: who is interrupting, how much and on which core.

With this you close module 2, and the whole journey is worth looking at. You started by opening up the process abstraction (02-01): its memory image, the task_struct, the states with their real codes, fork with copy-on-write, execve, zombies and the cost of a context switch. Then you saw how the CPU is shared between them (02-02): bursts, conflicting criteria, the classic algorithms worked out by hand and CFS's fair virtual time with nice. Then the second great resource, memory (02-03): logical and physical addresses, the MMU, fragmentation with numbers and why paging wins; and its full development (02-04): translation, multilevel tables, the TLB, page faults, replacement, thrashing, mmap and the OOM killer. Then storage (02-05): the physics of the disk and the SSD, request scheduling and RAID. And finally devices (02-06): making them uniform, /dev, udev and the three ways of talking to the hardware, which this lesson has developed to the end.

The seven lessons answer the same question from different angles: how a scarce resource is shared among many that want it. CPU, memory, disk, devices. And in all of them the same pattern has appeared: an abstraction that hides the complexity, a hardware mechanism that enforces the rules, and an operating system policy that decides.

But we have taken something big for granted. Every time two processes shared a page with MAP_SHARED, every time a softirq and a process touched the same queue, every time two meteo-api workers wrote into /dev/shm/meteora-cache, we have said "this requires synchronization" and moved on. The time has come to pay that debt. What exactly happens when two flows of execution touch the same piece of data at once? Why can counter++ lose increments? And how do they coordinate without wrecking the performance that has cost so much to achieve?

That is Module 3: Concurrency, and it starts in Concurrency Concepts.