It is 03:12. The phone buzzes: "The Meteora API is responding slowly. Customers have been complaining for a few minutes." There is no more information, nobody knows what changed, and the person who wrote the aggregator is on holiday. You have SSH access to meteo-01 and the rest is up to you.

This lesson is unlike every previous one. It introduces no new concepts: it uses all the ones you have learned. You are going to walk through a complete incident, with the real output of every command, the hypotheses that get ruled out, the two that turn out to be true and a third that appears without being looked for and that completely changes the nature of the problem. You will see how module 2's theory about disk access patterns turns into a line of iostat, how module 3's lock hierarchy turns into a thread hung in /proc/<pid>/stack, and how module 5 forces the performance investigation to stop dead.

At the end we will close the course: a map of what we have covered, the paths that open up and how to keep practicing.

1. The response playbook
2. First phase: the initial 60 seconds
3. Second finding: the aggregator and the access pattern
4. Mitigation versus real fix
5. Third finding: threads in state D that never progress
6. The deadlock, its fix and the proof
7. Fourth finding, unexpected: the incident changes its nature
8. A blameless post-mortem
9. Final exercises
10. Closing the course

Before typing a single command, three decisions. They cost thirty seconds and they change the outcome of the incident.

First: bound the symptom with data, not with impressions. "It's slow" gives you no way to check whether anything improves. You need a number and a moment in time.

Compare it with the baseline for an ordinary Tuesday (p50=0.031 p95=0.104 p99=0.198) and you already have the symptom quantified: the p99 has grown thirtyfold. Repeating the query minute by minute, the degradation begins between 03:04 and 03:06. That moment is your anchor: everything you investigate gets correlated against it.

Second: decide whether to mitigate or investigate first. They are goals in tension. Mitigating restores the service but destroys evidence: if you restart meteo-api, the state that explains the failure disappears. Investigating preserves the proof but prolongs the outage. The criterion is impact:

At 03:12 the API is responding, with bad latency but no mass errors. You decide to investigate, with an explicit time limit: fifteen minutes.

Third: write everything down with timestamps. Open an incident logbook and save every output:

Why this matters so much. Three hours from now you will not remember whether aqu-sz was 18 or 27, and the post-mortem depends on those numbers. What is more, if the incident turns out to be a security matter —as it will— the logbook becomes part of the chain of custody from 05-04.

You run the whole 07-03 checklist, in order, skipping nothing.

What it rules out. Nothing yet, but it orients you: a load of 28 on an 8-core machine, with the 1-minute average far above the 15-minute one. The problem is recent and growing. There was no reboot (41 days of uptime), so we rule out a failed boot.

What it rules out. There are no OOM kills, no device errors, no degraded RAID, no segfault. That eliminates in one stroke the three most frequent catastrophic causes. The recovery done line at 02:41 comes from a RAID resync that has already finished: it could have been the cause, but it ended 24 minutes before the symptom. Noted as not entirely ruled out, because every temporal coincidence deserves a review.

Here is the first big finding. r=1 (a single process ready to run) against b=31 (thirty-one processes blocked in uninterruptible I/O). si/so at zero: it is not memory. us+sy around 6% with wa at 89%: it is not CPU. The load of 28 is entirely explained by the processes in state D from 02-01, which on Linux —and only on Linux— count towards the load average.

What it rules out. The %iowait is spread evenly across the eight cores, so it is not one saturated core nor a badly distributed interrupt (02-07). And %steal at zero rules out the hypervisor (06-01): nobody is stealing CPU from us.

What they rule out. An available of 6.1 GB and memory pressure of zero: memory is definitively out. The established connections are in the normal range. And PSI is conclusive: an io full avg10 of 71.6% means that over the last ten seconds the machine has spent seven out of every ten unable to make progress because it was waiting on the disk, while CPU pressure is residual.

Conclusion of the first phase, at 03:17: the bottleneck is disk I/O. Three minutes have been enough, and CPU, memory, swapping, network, the hypervisor and a hardware failure have all been ruled out with data. Write it down and carry on.

A full reading of this output, which is the heart of the diagnosis. The bandwidth is derisory: 16 MB/s, something any twenty-year-old disk would deliver without breaking a sweat. But there are 4,102 operations per second, and rareq-sz is pinned at 4.00 KB: minimum-size requests. rrqm/s is zero, meaning the I/O scheduler (02-05) cannot merge anything, which only happens when the requested blocks are not contiguous. It is the exact signature of random reads.

A RAID 1 of two mechanical disks delivers on the order of 150-200 random IOPS per disk, some 300-400 on reads adding both together because the mirror spreads the reads. We are asking for more than ten times that capacity. Hence aqu-sz=27.4 —twenty-seven requests waiting on average, USE's saturation— and r_await=52 ms. The %util of 99.9% is true but not very informative: what hurts is the queue.

Who is causing it?

What it proves. The aggregator is reading almost 16 MB/s —the device's entire throughput— and is accumulating an enormous iodelay: it is the one saturating the disk. meteo-api barely reads, but its iodelay of 41 shows that it is waiting too, and that wait is the latency the user suffers. We have a culprit and a victim.

It started at 03:00:14 from its timer and has been running for 17 minutes, when it normally takes two. That fits perfectly with the symptom starting at 03:05.

The decisive finding. A single day's file takes up 17,280,000 bytes, about 16.5 MiB, and should fit in a handful of contiguous extents. It has 5,314. And read_bytes says the aggregator has read 17.9 GB from disk in order to process a 16.5 MB file: it is re-reading the same file more than a thousand times, or accessing it in a completely disordered way.

Why the access pattern matters so much, which is the theory of 02-05 and 04-05 made flesh. On a mechanical disk, a sequential read of 16.5 MB is a single head positioning followed by a continuous transfer: about 0.15 seconds. That same 16.5 MB read as 4,200 random 4 KB requests is 4,200 seeks of about 10 ms each: 42 seconds, nearly three hundred times more, with exactly the same bytes read. The disk is not slow; the pattern is bad. And the fragmentation into 5,314 extents turns even a "sequential" read of the file into something resembling random access, because the logically contiguous blocks are physically scattered: it is the consequence of the ingestor appending to the file over 24 hours while the file system allocated space wherever it could.

It is 03:24 and the service has to be restored. Always keep the two things apart.

Immediate mitigation (minutes, reversible, does not fix the cause):

What it does and why it works. ionice -c 3 moves the process into the I/O scheduler's idle class: its requests are only served when there is no other request outstanding. meteo-api stops competing on equal terms and its latency should drop immediately.

Verify, always, that the mitigation works:

Partial result. r_await drops from 52 to 11 ms, aqu-sz from 27 to 6, and the API's p99 goes from 6.1 s to 0.94 s. An enormous improvement… but the baseline was 0.198 s. Something is still wrong, and that residue is what will lead to the third finding. Noting it is essential: the classic incident trap is to declare the case closed as soon as things improve enough.

Lasting mitigation, applying the cgroups from 06-02 at the unit level, so that it does not depend on somebody running ionice by hand:

What it achieves. IOSchedulingClass=idle makes the ionice change permanent; IOWeight=10 against meteo-api's IOWeight=200 splits the disk twenty to one when both compete; and IOReadBandwidthMax puts a hard ceiling translated into io.max in the service's cgroup. It is the same kernel mechanism that limits a container, applied to a native service. An important note: MemoryMax=1G also bounds the damage of a possible leak, killing only the aggregator instead of letting the OOM killer pick a victim (02-04).

The real fix, which is not done at three in the morning but is recorded as a post-mortem action:

It is 03:31. The disk is no longer saturated, but the p99 is still at 0.94 s, almost five times the baseline. You go back to meteo-api.

What it reveals. ps -eLo lists threads (-L), not processes: without that option you would see none of this (03-02). Three threads are in state D and the wchan field —the kernel function they are sleeping in— says exactly what they are waiting for: flock_lock_inode_wait, that is, a file lock from 04-04. And two more are in futex_wait_queue, waiting on a user-space mutex (03-04).

Confirmation from the kernel. The thread's in-kernel stack confirms it is inside the flock() call and not somewhere else. It is the same technique you learned in 03-06 for diagnosing a process stuck in D.

The full picture. /proc/locks is the kernel's lock table, and there are two protagonists here: PID 1834 (meteo-api) holds the cache lock (cache.lock), and PID 9127 (aggregator) holds the data file lock. Each is waiting for the one the other holds.

And here is the root cause. The user-space stack gives away the real acquisition order: refresh_cache_from_disk() takes the cache lock first and then asks for the file one. But the hierarchy agreed in 03-06 for the whole system is:

configuration → cache → file → log

Hold on: that code respects the hierarchy (cache before file). The one violating it is the other end. Looking at the aggregator:

Confirmed: a deadlock caused by a hierarchy violation. The aggregator took the file lock first and is now asking for the cache one; meteo-api took the cache one and is asking for the file one. It is the circular wait cycle, Coffman's fourth condition, in its purest form and with only two participants. gdb -batch with -ex 'thread apply all bt' is non-destructive if used carefully, but it stops the process while it runs: use it briefly and knowing that you are adding latency.

An important note on why this had not been seen before: with the aggregator finishing in two minutes, the overlap window was minimal. Once it stretched to 17 minutes because of the disk saturation, the probability of a collision shot up. The first problem uncovered the second, which had been latent for months. It is a common pattern: rare deadlocks become frequent as soon as something slows the system down.

Immediate mitigation, because the cycle does not break by itself:

Why kill the aggregator and not meteo-api. It is the recovery-by-termination of 03-06, choosing the victim on two criteria: the aggregator is re-runnable —its timer will launch it again, and with Persistent=true the run is not lost— whereas restarting meteo-api would cut off 1,188 established connections. What is more, SIGTERM rather than SIGKILL gives it the chance to close cleanly, avoiding exactly the truncated files that the size % 24 != 0 check from 07-01 would detect.

Service restored at 03:44: a p99 of 0.204 s, indistinguishable from the 0.198 s baseline. And now for the real fix, which is a code change:

And the proof that it no longer happens, which is the part almost everybody skips:

Why this way. The first test reproduces the scenario that was rare in production; if the deadlock persisted, it would show up within a few attempts. The second is a cheap continuous check that can be turned into a metric. The third is the most valuable in the long run: centralizing every lock acquisition in a single module that always takes them in the hierarchy's order turns the discipline into something the compiler and a review rule can police, instead of depending on every programmer remembering the agreement. It is 03-06's prevention put into practice.

It is 03:52. The service is fine and you are gathering material for the post-mortem. You check whether the aggregator left anything behind:

You stop. That file has the setuid bit (rws), belongs to root, has a hidden name and a timestamp of 02:51. Nothing in Meteora's system creates that. And then you check the other thing that had struck you as odd:

There is an eleven-minute gap (02:47-02:58) in a log that writes thousands of lines per minute. A service that was blocked would leave fewer lines, not zero. A clean, exact gap in a log file is, until proven otherwise, tampering.

At this point the incident stops being a performance one. Two independent indicators —a root setuid binary that appeared at 02:51 in /tmp and a gap in the logs surrounding it— point to a possible compromise. From here on, everything you learned in module 7 becomes subordinate to module 5.

The first thing is what you do NOT do:

• Do not run the binary, not even with --help or on a "test" machine. It is root setuid.
• Do not delete it. It is the main piece of evidence.
• Do not reboot the machine. You would lose all the memory, the connections and the processes, which is precisely the most valuable material.
• Do not carry on "investigating the performance". Every command you run modifies access times, journal entries and shell history, and contaminates the scene.
• Do not raise the alarm through a channel that might be compromised. If the attacker has access, they will read your messages.

Preserve in order of volatility (05-04), from the most ephemeral to the most durable:

What each one contributes. stat gives the inode's three timestamps without opening the file. The SHA-256 hash makes it possible to prove later that the evidence was not altered: it is the technical foundation of the chain of custody. last -F shows the logins with full dates. ausearch queries auditd —if it was running, which is why you install it before you need it— for the executions in that window. And the find looks for other recent setuid files across the whole system, because an attacker rarely leaves just one.

Contain without destroying, in this order:

1. Isolate the network while keeping the machine powered on: nftables rules that allow only your administrative access. Powering it off destroys the memory; disconnecting it entirely may also tip off the attacker.
2. Do not change credentials yet if that would warn the intruder, unless instructed by whoever is coordinating the response.
3. Escalate immediately: the security lead, the service owner, management, and legal counsel and compliance. If personal data was accessed, in the EU the GDPR sets notification deadlines of 72 hours that start running from the moment the fact becomes known, and that decision is not made by whoever is at the console at four in the morning.
4. Move to an out-of-band communication channel and document who knows what, and since when.

Explicit warning. This narrative is teaching material. A real incident response must follow your organization's formal procedure and must have legal advice. Decisions about preserving evidence, notifying authorities and customers, public communication and any eventual criminal complaint have legal and contractual consequences that go beyond the technical. If your organization does not have that procedure written down, writing it before the next incident is worth more than any tool.

And why the performance investigation stops here. For three solid reasons. First, evidence contamination: every command alters the scene and can render the forensic analysis useless. Second, a change of priority: high latency costs money, a compromise can cost the data of thousands of people. And third, a change of hypothesis: if there is an intruder, everything diagnosed becomes suspect —did the aggregator slow down on its own, or did somebody force it? does the log gap cover the moment the binary was placed?—. You do not know whether the performance problem was the cause, the consequence or the smokescreen. Investigating performance on a possibly compromised system is building on sand.

A blameless post-mortem starts from a proven premise: people act reasonably given the information they have at the time. If the goal is to point at somebody, people hide information and the same failure happens again. If the goal is to understand the system, you learn.

Timeline (all times CEST on 2026-08-31):

Root cause versus contributing causes. The distinction is not academic: it determines where the effort goes.

• Root cause of the degradation: the aggregator reads the data with a 4 KB random pattern —re-reading 17.9 GB to process 16.5 MB— over files fragmented into more than 5,000 extents, which exceeds the RAID 1's IOPS capacity by an order of magnitude.
• Contributor 1: there were no I/O limits in the aggregator's unit, so it could consume 100% of the disk competing as an equal with the user-facing service.
• Contributor 2: the aggregator violates the agreed lock hierarchy. Latent for months, it surfaced once its run stretched out. It was not the cause, but without it the service would have returned to normal at minute 12 instead of minute 32.
• Contributor 3: there was no alert on I/O pressure. The warning came from the customers, not from the system, seven minutes late.
• Contributor 4: /tmp was mounted without noexec or nosuid, which allowed a setuid binary to be executable there.
• Contributor 5: the logs existed only on the machine, so the gap cannot be checked against any external copy.

Concrete preventive actions, each with an owner and a deadline:

Notice the pattern: not one action is "be more careful". They are all changes to the system —limits, alerts, mount options, code structure— that make the failure impossible or make it detect itself. That is the test of whether a post-mortem was worth anything.

Final tips for the on-call shift: always start with dmesg; quantify before touching anything and measure again afterwards; put a time limit on the investigation phase and respect it; speak up early even when you do not have the answer, because people tolerate communicated uncertainty far better than silence; and distrust the first explanation that fits, because real incidents, like this one, usually have more than one cause.

Every night between 02:00 and 04:00, meteo-api stops responding for a few seconds. systemctl status says active (running) when you look at it in the morning, but Main PID has changed. The data collected:

Diagnose it with the method from 07-03: who caused the problem, why the one that died was the one that died, and what mitigation and what real fix you would apply.

After editing /etc/meteora/meteora.conf to add a new cache path, meteo-api will not start:

Run by hand as meteora, the binary starts with no problem at all. Explain the contradiction and give the correct solution, plus two incorrect ones that must be avoided and why.

Solution 1

Diagnosis. The indicators are unambiguous and none of them is about I/O: an available of only 402 MB, si=1,204 and so=1,890 pages per second simultaneously —the system is pushing pages in and out at the same time, which is the definition of thrashing (02-04)— and a memory pressure of 58% (some) with 31% full: for almost a third of the time the whole machine makes no progress.

Who caused the problem and who died are not the same. The first line of dmesg says it literally: aggregator invoked oom-killer, that is, it was the aggregator's memory request that exhausted the system and triggered the mechanism. But the one chosen was meteo-api, with 2.9 GB of anon-rss. The reason lies in the oom_score, which is essentially proportional to resident memory: the OOM killer kills the biggest process, not the guilty one. And since meteo-api runs under systemd with Restart=on-failure, it restarts by itself, which explains the active (running) with a different Main PID in the morning and the fact that nobody noticed.

The 02:00-04:00 window coincides with the aggregator's nightly runs; the mechanism is memory accumulation in the aggregation process —most likely it loads the entire history into RAM instead of processing it in chunks.

Mitigation (tonight, without touching code):

With this, when the aggregator goes past 1 GB, the cgroup's OOM killer will kill it and only it, without touching the rest of the system; MemoryHigh adds an earlier step in which the kernel throttles it and reclaims pages aggressively before the hard limit is reached. It is exactly the mechanism from 06-02. It is also worth protecting the victim with MemoryMin=512M in meteo-api.service, which reserves memory the kernel will not reclaim from it.

Real fix: process the history in chunks with bounded, constant consumption, instead of loading it whole. And verification: track the aggregator's RSS during a run (while true; do awk '/VmRSS/{print $2}' /proc/$(pgrep -x aggregator)/status; sleep 10; done) to check that it stabilizes instead of growing linearly. Preventive alert: memory some avg60 > 20% for 5 minutes, which would have warned weeks before the first customer did.

What you must not do: lower vm.swappiness (the shortage is not of swap, it is of memory), add RAM without understanding the growth (that only delays the problem), or tweak meteo-api's oom_score_adj so it does not get chosen (you would just move the death to another innocent process).

Solution 2

The contradiction is only apparent, and its explanation is the unit's hardening. By hand, the binary runs with the normal file system and meteora can write wherever its permissions allow. Under systemd, the unit has ProtectSystem=strict, which mounts the entire system tree read-only inside the service's mount namespace, with the only exceptions being those declared in ReadWritePaths=: /var/lib/meteora, /var/log/meteora and /run/meteora. The new path, /var/cache/meteora, is not on that list, so the process sees a read-only file system and fails with EROFS. It is the isolation mechanism from 05-03 working exactly as it should.

The correct solution, which additionally delegates to systemd the creation of the directory with the right owner and mode:

CacheDirectory=meteora creates /var/cache/meteora owned by meteora:meteora, adds it automatically to the writable paths and manages it as part of the service's life cycle. The acceptable alternative, if the directory already exists and is managed by another process, is to add ReadWritePaths=/var/cache/meteora.

Two incorrect solutions and why:

1. Removing ProtectSystem=strict. It solves the symptom by disarming the protection: the service becomes able to write anywhere in the system, /etc and /usr included. You trade a five-line configuration problem for a permanent increase in attack surface, in precisely the service exposed to the Internet. The systemd-analyze security score would reflect it immediately.
2. Running the service as root (or giving the directory chmod 777). It throws away the whole chain of decisions made through the course: the shell-less account, UID 990, CAP_NET_BIND_SERVICE instead of root, NoNewPrivileges. And 777 would let any user on the system —including a compromised process— tamper with the service's cache.

General lesson: when a service works by hand and fails under systemd, the difference is almost always in the environment (PATH, working directory, variables) or in the hardening (ProtectSystem, ReadWritePaths, SystemCallFilter, capabilities). And the right answer is almost always to declare the specific exception, never to disable the protection.

You have resolved a complete incident, and in doing so you have used the entire course. It is worth looking at the map, because every piece of theory has ended up turned into a diagnostic tool.

From module 1 came the idea that the operating system is an extended machine and a resource manager, and that everything goes through system calls; without that, strace, wchan and /proc/<pid>/stack would be magic. From module 2 came the process states —D and its weight in the load average—, the disk access patterns that explain why 16.5 MB can take 42 seconds, the OOM killer choosing by size rather than by guilt, and RSS versus VSZ for detecting a leak. From module 3 came threads, flock, the futex and, above all, the lock hierarchy whose violation produced the deadlock and whose centralization prevents it. From module 4, inodes and /proc/locks, the extent fragmentation filefrag revealed, and the atomic writing that prevents truncated files. From module 5, everything that happened from 03:52 onwards: the setuid binary, the gap in the log, the order of volatility, the chain of custody, noexec on /tmp and the obligation to escalate. From module 6, the cgroups that limited the aggregator with io.max and MemoryMax. And from module 7, the shell that ran it all, systemd that governs it and the method that gave the investigation its order.

That is the central message of the course: theory is not a toll you pay before practice, it is what lets you interpret what you see. Two people run iostat -x and see the same numbers; only one knows that a rareq-sz of 4 KB with rrqm/s at zero means random access, that the queue matters more than the utilization, and that a RAID 1 doubles reads but not writes. The difference is not in the command, it is in modules 2 and 4.

The paths that open up from here:

They all share the same foundation: the one you have just finished.

How to keep practicing, which is the only thing that makes this stick:

• Build a lab virtual machine and break it on purpose. Provoke an OOM, saturate the disk with fio, create a deadlock with two scripts and flock, delete a unit file and repair it. Nothing teaches like fixing something you broke yourself.
• Read /proc out of curiosity. Every file in /proc/<pid>/ is a window onto a kernel structure. Spend an afternoon walking through status, maps, io, limits, stack, fd/ and environ of a real process: you will understand more than from many chapters.
• Redo the course's exercises on your own machine, changing the numbers. The ones in module 3 and the labs in 07-03 pay off the most.
• Read logs even when nothing is happening. Getting familiar with what a normal journalctl looks like is what will let you spot the abnormal in three seconds.
• Write your own post-mortems, even when the incident is domestic and you are the only reader. Putting the timeline and the root cause in writing is what turns an experience into knowledge.

We began by defining the operating system as an extended machine that hides the complexity of the hardware. We end at four in the morning on a real server, reading that complexity through the windows the system itself offers us: /proc, the journal, the kernel's counters. Between the two points there are seven modules, but really there is a single idea repeated: the operating system is not a black box. It is a program, written by people, that makes comprehensible decisions about limited resources, and that leaves a trail of every one of them. Learning to read that trail is what separates somebody who uses a computer from somebody who understands it.

You know how to do it now. The next time the phone rings at three in the morning, you will not know the answer —nobody does— but you will know how to find it: quantify the symptom, rule things out with data, look for saturation rather than utilization, distrust the first explanation that fits, and write everything down so the next person has an easier time.

Thank you for making it this far. Now close this, open a terminal and break something.