meteo-01 starts on its own, meteo-api is active (running), the aggregator's timer fires punctually and systemctl list-units --failed returns nothing. And yet users say the API is slow. None of the previous lesson's tools answers the question "why?", because they all answer "is it running?" and not "is it running well?".

This lesson teaches the second one, and it does so in a deliberate order: method first, tools afterwards. The reason is that most failed diagnoses do not fail through ignorance of a command, but through jumping to conclusions: someone sees a high load, blames the CPU, adds cores and the problem persists because it was really the disk queue. A method prevents that. You will see two named methodologies —USE and RED—, the first-60-seconds checklist you will always run the same way, and then, resource by resource, what each number returned by top, free, iostat, ss and vmstat really measures, along with the interpretation traps that fool almost everybody. We will finish by moving up a level: from watching one machine to instrumenting a fleet, and with a lab in which you will provoke each type of saturation so you can see it with your own eyes.

The integrated case study, where all of this is applied to a real middle-of-the-night incident, is the next and final lesson of the course.

1. Symptom, cause and baseline
2. Two methodologies: USE and RED
3. The first-60-seconds checklist
4. CPU: what each number really measures
5. Memory: why free -h confuses everybody
6. I/O: iostat -x field by field
7. Network: queues, retransmissions and latency
8. Resource pressure: PSI
9. Deep observation and its cost: strace, perf, eBPF
10. Logs as a source of diagnosis
11. From one-off observation to continuous monitoring
12. Useful alerts, SLIs and SLOs
13. Capacity planning
14. Guided lab: provoking and observing each kind of saturation

A symptom is what is perceived: "the site is slow", "the reports arrive late", "the application returns 502 errors". A cause is a concrete, verifiable mechanism: "the aggregator is doing 4 KB random reads and the RAID 1's queue is at 18 outstanding requests with a 40 ms average wait". Between the two there is a chain of findings, and the work of troubleshooting consists of walking it by ruling out rather than guessing.

Three rules that avoid most mistakes:

• Quantify the symptom before touching anything. "It's slow" is not data. "The p99 of /v1/readings has gone from 80 ms to 4.2 s since 03:05, with the same request volume" is, and it also gives you a start time to correlate everything else against.
• One hypothesis, one check. Change one thing at a time. If you apply three mitigations at once and things improve, you will not know which one worked and you will not be able to write an honest post-mortem.
• Take timestamped notes. What you were looking at, what you saw and what you did. In a long incident, memory fails and the command outputs are unrepeatable.

And above everything else there is the baseline. An isolated number means nothing: a load average of 6 can be normal on a 16-core machine and catastrophic on a 2-core one. A disk %util of 60% may be perfectly usual at midday. Without knowing how meteo-01 behaves on an ordinary Tuesday, any measurement during the incident is noise. That is why the best time to prepare for a diagnosis is when everything is fine: if you do not yet have continuous monitoring, at least save a periodic snapshot.

What it does. It dumps into a single file, with an ISO timestamp, the load, a vmstat sample (the first record from these tools is an average since boot and must be discarded, hence the tail -1), the memory, the RAID's activity and the socket summary. Run from a systemd timer every 5 minutes, in a week you have a reference to compare against. It costs a few kilobytes a day and it has saved more incidents than plenty of dashboards.

USE (Brendan Gregg) is applied resource by resource and answers "which component of the system is the limit?". For each one you look at three things:

• Utilization: what fraction of the time the resource is busy.
• Saturation: how much work is waiting in a queue because the resource cannot keep up.
• Errors: failures counted by the resource.

Saturation is the key metric and the most ignored one. A disk at 100% utilization with a queue of 1 is working comfortably; the same disk at 100% with a queue of 30 is drowning. Utilization has a ceiling (100%) and therefore stops being informative the moment it is reached; saturation has no ceiling and keeps growing with the severity of the problem.

RED is applied to the service and answers "are the users having a bad time?": Rate (requests per second), Errors (how many fail) and Duration (how long they take, in percentiles). It is the view from outside; USE is the view from inside. They are used together: RED detects and bounds the problem, USE locates the guilty resource.

The full flow, which is the one you will follow in the next lesson:

The table applied to meteo-01:

This sequence is run always the same way, in the same order, without thinking. Its value is not that it finds the cause, but that in one minute it rules out 80% of the possibilities and tells you where to dig deeper.

What to look for in each one, which is what actually matters:

1. uptime — the three load averages (1, 5 and 15 minutes). What is informative is not the number but the trend: if it reads 24.1, 8.4, 3.2, the problem is starting right now; if it reads 3.2, 8.4, 24.1, it is already receding. It also confirms whether the machine has rebooted recently.
2. dmesg -T — this is the first thing to look at because it contains the failures no other tool shows you: an OOM killer kill, a disk with read errors, a degraded RAID, a connection-table overflow. A single message here can end the diagnosis in 10 seconds.
3. vmstat 1 5 — the panorama. r is the run queue (processes ready and waiting for CPU); b those blocked in uninterruptible I/O; si/so the swapping; us/sy/id/wa the CPU breakdown; cs the context switches from 02-01.
4. mpstat -P ALL 1 3 — it distinguishes "the whole machine is saturated" from "one core at 100% and fifteen idle", which is the symptom of a single-threaded program or of a badly distributed interrupt (02-07).
5. pidstat 1 3 — unlike ps, it shows consumption per interval, not accumulated since boot. It is the difference between "this process has used 40 hours of CPU over two months" and "this process is using 180% right now".
6. iostat -xz 1 3 — -z omits devices with no activity, leaving only the interesting ones.
7. free -m — with the caveat from the memory section: look at available, not at free.
8. ss -s — the socket summary; a jump in timewait or in synchronized connections points at the network or at the client pattern.
9. sar -n DEV 1 3 — traffic per interface, to rule out link saturation.
10. PSI — the direct answer to "how much time is being lost waiting for each resource?".

And always, at the end, the logs: failed units and recent errors in the journal.

The load average is not CPU utilization. It is the most widespread interpretation error there is. On most UNIX systems, the load counts processes in the R state (running or runnable). On Linux, and only on Linux, it also counts those in the D state, that is, in uninterruptible I/O wait (02-01).

The practical consequence is enormous: on meteo-01, with 8 cores, a load of 24 can mean three completely different things —24 processes fighting for CPU, or 2 using CPU and 22 waiting on the disk, or any mixture— and the right answer is the opposite in each case. Adding CPU to the second scenario fixes nothing. That is why the load tells you that something is happening, never what.

What the combination reveals. A load of 24 with 88% %iowait and 4% %usr is conclusive: the CPU is practically idle and the processes are piling up waiting on the disk. The ps filtered on D names the culprits. If instead you saw %usr at 95% and %iowait at 0, the problem would genuinely be computational.

How to read the header, which is where nearly all the information is. The task line already hints at the diagnosis: 27 processes in the "stopped" state —which top lumps together with the uninterruptible ones— with only one running is an enormous anomaly. htop presents the same thing interactively, with one bar per core (handy for seeing at a glance whether the load is spread out), the process tree with F5 and the ability to sort by any column; its real advantage is that it makes visible in a second what in top you have to go looking for.

The fields of the CPU line, one by one:

%wa deserves a nuance almost nobody explains: it is not time "spent" on I/O, it is idle time with I/O outstanding. If the CPU had other work to do, it would do it and %wa would drop without the I/O improving in the slightest. That is why a low %wa does not rule out a disk problem on a busy machine: you have to look at iostat.

%st is the metric that only exists on virtual machines, and it is the one that explains a recurring mystery: "my process takes twice as long and the CPU is at 50%". If st is 15%, the hypervisor is taking one in every seven cycles away from you and there is nothing you can change inside the guest.

Why the per-thread breakdown matters. A process at 100% on an 8-core machine can be a saturated single-threaded program —a real ceiling, with nothing more to squeeze out without changing the code— or a barely busy multi-threaded process. pidstat -t or top -H tell them apart in two seconds, and that fact completely changes the recommendation.

Almost everybody reads "free: 412 MB" and panics. That is the wrong reading. Linux uses all the spare RAM as a page cache (02-03), because free memory is wasted memory: by keeping recently read files there, it avoids going to disk. That cache is reclaimable instantly: if a process asks for memory, the kernel drops clean cache pages and hands it over.

The column that matters is available (MemAvailable), which is the kernel's estimate of how much memory a new process could get without causing swapping. Here, 4,102 MB: the machine is comfortable despite the "412 free".

How to read the swapping. swpd (how much is in swap) can be high with nothing wrong: those are pages that were evicted days ago and nobody has asked for since. What indicates an active problem are si and so, the pages coming in and going out per second. A sustained so of 2,048 pages/s is 8 MB/s being written to swap; if there is also a high si at the same time, the system is thrashing (02-04): it spends more time moving pages than working, and latency shoots up by two orders of magnitude.

Detecting memory leaks. A leak is not visible in a snapshot: it is visible in a film. Track the RSS —physical memory actually occupied, as opposed to VSZ, which is only reserved address space (02-04)— over time:

How to interpret it. A healthy service stabilizes its RSS after warm-up; a leak draws a rising straight line that never bends. If meteo-api's RSS climbs 40 MB every hour, steadily, in 50 hours it will reach the MemoryMax=2G in its unit and systemd will kill it. The advantage of having set that limit back in 07-02 is that the leak kills only the guilty service instead of letting the system's OOM killer pick a victim.

The OOM killer's trail, which you need to recognize from memory:

What that line tells you. It names the chosen process, its UID (990, that is meteora) and its anon-rss at the moment it died: 2.9 GB. Remember from 02-04 that the OOM killer chooses by oom_score, which is mostly proportional to memory used, so it usually kills the biggest process, not the guilty one. A typical and disconcerting symptom is that the database dies because some sloppy script asked for all the RAM.

Diagnosis of the example. A rareq-sz of 4.00 KB with rrqm/s at zero says these are random single-block reads: there is nothing to merge because the blocks are not contiguous. 3,891 reads/s of 4 KB is only 15.5 MB/s of bandwidth —derisory— but nearly 4,000 IOPS, which for a RAID 1 of mechanical disks (about 150-200 IOPS per disk) is an order of magnitude above its capacity. Hence the aqu-sz of 18 (eighteen requests waiting on average) and the r_await of 41 ms. The cause is not "the disk is slow": it is the access pattern, exactly what was explained in 02-05.

Why %util misleads on NVMe. The field is computed as the percentage of time during which there was at least one request in flight. On a mechanical disk, with a single head, that genuinely equals "busy". A modern NVMe serves tens of thousands of IOPS in parallel spread across multiple queues: it can sit at 100% %util while using 5% of its real capacity. On SSDs and NVMe, the trustworthy metrics are await and aqu-sz, not %util.

What each one is for. pidstat -d attributes the I/O to specific processes, which is the step that turns "the disk is saturated" into "the aggregator is saturating the disk"; its iodelay column measures the time the process has spent blocked waiting. filefrag counts a file's extents (04-05): if a 17 MB file lives in 3 extents, reading it whole is practically sequential; if it lives in 4,000, even a full read behaves like random access.

What Meteora's RAID 1 saturation looks like. Notice that in the output above md0 receives 3,891 reads/s and each physical disk serves about 1,945: RAID 1 spreads the reads across the two copies, which doubles the available read IOPS, but does nothing at all for writes, because every write must go to both disks. That is the fundamental asymmetry of the mirror, and it explains why RAID 1 is good for availability and somewhat for reads, but never solves a write problem.

What each thing means, and there is a subtlety here that confuses a lot of people. On a listening socket, the columns change meaning: Recv-Q is the number of already established connections waiting for the application to call accept() and Send-Q is the maximum size of that queue (the Backlog=2048 from the unit in 07-02). A sustained Recv-Q of 129 means meteo-api is not accepting connections as fast as they arrive: the problem is not the network, it is that the application is busy or blocked. If the queue fills completely, the kernel starts dropping SYNs and the ListenOverflows counter grows; the client sees a connection that does not answer and retries, amplifying the problem.

Retransmissions (TcpRetransSegs) indicate packet loss: a sustained rate above 0.1% of total segments points at congestion or a faulty link. And in ip -s link, the receive errors and dropped columns tell a physical fault (cable, speed negotiation) apart from a drop caused by a full queue inside the kernel itself.

Latency versus bandwidth, a distinction that decides many designs: bandwidth is how many bytes per second fit; latency is how long the first one takes to arrive. A 10 Gb/s link with a 200 ms round trip is magnificent for transferring a 50 GB file and terrible for an API that makes 30 chained queries, because those 30 round trips are an irreducible 6 seconds: no amount of extra bandwidth brings them down. When meteo-api responds slowly, the right question is whether it is slow to start responding (latency, dependencies, locks) or slow to finish (volume, bandwidth).

PSI (Pressure Stall Information, Linux 4.20+) is probably the most useful metric to have appeared in the last decade, and it answers exactly the question the others dodge: how much time is being lost because of each resource.

How to read it. some is the percentage of time in which at least one task was blocked waiting for that resource; full is the percentage in which every runnable task was, that is, time in which the whole machine made no progress. The three numbers are moving averages over 10, 60 and 300 seconds, which gives you a trend without needing to sample.

An io full avg10 of 61% means, literally, that over the last 10 seconds the machine has spent six seconds out of every ten unable to do anything because it was waiting on the disk. Compare that with what the classic metrics would say at the same moment: the CPU looks idle (high %id), the memory looks fine and only %wa hints at anything. PSI says it unambiguously and, above all, in units of impact —time lost— rather than in units of resource. That is why it is an excellent basis for alerting: io full avg60 > 20% is a threshold that means the same thing on any machine, with any disk.

PSI also exists per cgroup (/sys/fs/cgroup/system.slice/meteo-api.service/io.pressure), which lets you answer "which service is suffering?" and "which service is causing the suffering?" separately.

When aggregate metrics are not enough, you go down to the detail. These tools answer "what exactly is this process doing?", but they cost, and you need to know how much.

A serious warning. strace works through ptrace, which stops the process on every system call entry and exit and adds two extra context switches for each one. The typical slowdown ranges from 10 to 100 times for a call-intensive process. On meteo-api in production, with thousands of requests per second, strace is not an observation: it is a self-inflicted outage. Use it with -c (which only aggregates), for a few seconds, on a non-critical process, or in a test environment. The rule is: strace to understand a program, never to measure one in production.

Why perf is viable. It intercepts nothing: it samples. At 99 Hz it takes 99 snapshots per second of where the program counter is and what stack lies beneath it; the typical overhead is below 1-2%. The 99 Hz frequency rather than 100 is a deliberate trick to avoid synchronizing with system timers that usually run at 100 Hz and would bias the sampling. The natural output of perf record is a flame graph: a drawing in which the horizontal axis is the proportion of samples —not time— and the vertical one is stack depth, so that wide plateaus point instantly at where the CPU is going. It is the fastest way there is to find a hot spot in code.

What eBPF adds. Verified programs that the kernel runs at instrumentation points, aggregating inside the kernel and returning only the result, without copying every event to user space. The first example counts openat calls per program; the second draws a histogram of block request sizes, which in Meteora's case would show at a glance the 4 KB spike that gives away the random access; biolatency gives the distribution of disk latencies, which reveals the long tails that an average hides. It is the natural evolution of everything above: the power of strace at a cost closer to perf's.

Metrics say what is happening; logs usually say why. With the tools from 05-04:

The technique with the highest payoff is temporal correlation. Take the exact instant the symptom started (03:05, according to the percentile you computed at the beginning) and look at everything that happened on the machine in that window, without filtering by service: a deployment, a scheduled job that kicked off, a logrotate, a disk reporting an error, a spike in connections. The cause is usually in that window, and often in a service other than the one showing the symptom. A suspicious gap in the logs —minutes with no entry at all in a file that is written every second— is itself a first-rate finding.

What it does. It counts how many entries there are per minute and shows the minutes with fewer than ten. In a service that logs thousands of requests per minute, a minute with two entries —or missing altogether— means the service was blocked, that the disk was not accepting writes… or that somebody tampered with the file, a possibility that must be taken seriously and that we will see in 07-04.

Everything above works when there is already an incident and you are sitting in front of it. That does not scale: the numbers need to be collected always, so you can look backwards and so that somebody is alerted without anybody watching.

What this proves. There is no magic: the exporter reads /proc/loadavg, /proc/meminfo and /proc/pressure/io —the very files you have been consulting by hand— and publishes them in a format that a time-series database scrapes every 15 seconds. Understanding this has a practical consequence: any number you know how to obtain with a command you can turn into a historical metric and into an alert, including the output of the verification script from 07-01.

Four practical decisions that determine whether the system helps or gets in the way:

• Sampling interval. At 60 seconds you will not see a 20-second spike; at 5 seconds you multiply the volume twelvefold. For infrastructure, 15 seconds is the usual sweet spot.
• Cardinality. Every distinct combination of labels is one time series. Labeling by station_id with 3,000 stations and 20 metrics is 60,000 series per instance: it is the most common way to bring a monitoring system down. Label by dimensions of bounded cardinality (service, endpoint, status code), never by free-form identifiers.
• Retention and resolution. High resolution for a short time (15 s for 15 days) and long-term aggregates (5 min for 2 years, for trends and capacity).
• Percentiles, not averages. If 99% of requests take 20 ms and 1% take 10 s, the average is 120 ms and describes nobody's experience. The p99 is the number that corresponds to the users who complain. Keep p50, p95 and p99, and remember that percentiles do not average: the mean of ten machines' p99s is not the p99 of the whole.

The difference between an alerting system and a source of noise fits in one sentence: alert on symptoms the user perceives, not on resource utilization.

An alert must meet three conditions: it is real (something is genuinely wrong), it is actionable (there is something the recipient can do) and it is urgent (it cannot wait until tomorrow). If any of them fails, it is not an alert: it is a dashboard, or a ticket.

Formalizing this gives you SLIs and SLOs. An SLI (indicator) is a measure of the user's experience: "percentage of requests to /v1/readings answered correctly in under 500 ms". An SLO (objective) is the committed level: "99.5% over a 30-day window". The practical consequence is the error budget: 0.5% of 30 days is about 3.6 hours of permitted breach a month. That budget turns arguments of opinion into decisions backed by data —if there is plenty of budget left, you deploy and experiment; if it has been spent, you freeze and stabilize— and it defines the alert that really matters: warn when the budget is being burned too fast, not when a resource crosses an arbitrary threshold.

Troubleshooting is reacting; capacity planning is not having to react. With long-term series you can answer questions such as when /var/lib/meteora will fill up, how many more stations the ingestor can take, or whether the RAID 1 will cope next summer.

An example with Meteora's numbers. Each day generates a file of 17,280,000 bytes, that is 16.5 MiB; per year, about 6.0 GiB. If the volume has 200 GB and 120 GB are already used, 80 GB remain, which at that rate is more than thirteen years: growth by number of days is irrelevant. But if the business plans to go from 500 to 3,000 stations, the daily volume multiplies by six (99 MiB/day, 35 GiB/year) and the margin drops to little more than two years… and, long before space, the problem will be IOPS: six times more concurrent reads on a RAID 1 we already saw saturated at 4,000 random IOPS.

Hence the two rules of capacity planning: project against the business driver (stations, users, requests), not against the calendar; and remember that resources do not run out at the same time —normally the bottleneck arrives through IOPS or latency long before space or CPU—. And bear in mind that latency does not grow linearly: by queueing theory, once utilization goes past 70-80%, waiting time shoots up, so planning for 95% utilization is planning an incident.

Explicit warning: do this on a disposable test virtual machine, never in production. Each exercise deliberately provokes a degradation; some of them can leave the machine unresponsive for a few minutes. Have a way to reboot it at hand.

Lab 1 — CPU saturation. In one terminal, stress-ng --cpu 8 --timeout 120s. In another, observe:

What you should see and why. The load climbs towards the number of workers; %usr near 100% and %iowait at zero; vmstat's r column (runnable processes) grows above the number of cores; and cpu some avg10 rises clearly while full stays low, because there is always somebody running. Contrast this with the I/O signature: high load with high %usr is CPU; high load with high %wa is disk. That comparison is the point of the exercise.

Lab 2 — Memory pressure and swapping. stress-ng --vm 2 --vm-bytes 80% --timeout 120s:

What you should see. available falls; si/so start moving once anonymous memory exceeds what fits; memory some rises before any other indicator does. If you push harder (--vm-bytes 95%), the OOM killer will arrive and you will see in dmesg the Killed process line with its anon-rss, which is exactly the trail you learned to recognize. Watch too how the cache (buff/cache) shrinks automatically to give memory back: the practical demonstration that it was not occupied memory.

Lab 3 — Random I/O saturation (the signature of Meteora's case):

What you should see and compare. rareq-sz pinned at 4 KB, rrqm/s near zero (nothing to merge), a high aqu-sz, an await of tens of milliseconds and %util close to 100%. Now repeat it with --rw=read --bs=1M: you will see vastly more bandwidth, a large rareq-sz, a high rrqm/s and a low await, with the same %util. That comparison is the central lesson of the lab and of the whole module: %util does not tell a comfortable disk from a drowning one, and the access pattern matters more than the device. With --direct=1 you also stop the page cache from masking your results.

Finish each lab by writing down the "signature" of each saturation: the exact combination of numbers you saw. That notebook is what will let you recognize the problem in three seconds when it happens for real.

Tips: always start with dmesg, because a single message can save you an hour; learn the 60-second list by heart and run it end to end even when you think you know the answer, because ruling things out has value; measure before and after every mitigation, with the same command; and save the outputs to a timestamped file (command | tee -a /var/tmp/incident-$(date +%s).log), because in the post-mortem those captures are unrepeatable.

Faced with a complaint about meteo-api being slow, you get this:

Say which resource is the bottleneck, what each line lets you rule out, and what your next command would be.

Two machines behave as follows. Determine which one has a real problem and what you would do in each case.

• A: free=210 MB, available=5,980 MB, swpd=2,100 MB, si=0, so=0, memory some avg60=0.4
• B: free=1,900 MB, available=1,980 MB, swpd=180 MB, si=1,840, so=2,310, memory some avg60=63.7

Rewrite these three alerts so that they meet the real, actionable and urgent conditions, and justify each change: (a) "Server CPU > 85% for 1 minute"; (b) "Swap memory is in use"; (c) "The meteo-api process is not running".

Solution 1

Bottleneck: disk I/O, with a 4 KB random read pattern. What each line rules out:

• Load 31 with %usr 3.1% and %idle 3.3%: rules out the CPU as the cause. With 8 cores, a load of 31 and almost no user time can only be explained by processes in state D, which on Linux count towards the load.
• %iowait 91.2% and %steal 0: confirms I/O waiting and rules out a hypervisor problem.
• available 6,120 MB, si/so at zero: rules out memory. The low free is normal because of the cache.
• vmstat with r=1 and b=27: the definitive proof. Only one process ready to run and twenty-seven blocked in uninterruptible I/O. It is the exact signature of disk saturation.
• rareq-sz=4.00 with rrqm/s=0: random 4 KB reads; the scheduler cannot merge anything because the blocks are not contiguous. 4,102 IOPS is an order of magnitude above what a mechanical RAID 1 delivers.
• aqu-sz=27.4 and await=52 ms: severe saturation, not merely high utilization. The disk is at 99.9% and has 27 requests waiting.
• io full avg10=74.28: over the last 10 seconds, the machine has made no progress for nearly three quarters of the time because it was waiting on the disk.

Next command: pidstat -d 1 5 (or iotop -oPa), to attribute that I/O to a specific process. After that, filefrag on the files involved and cat /proc/<pid>/io, to understand whether the random pattern comes from the program's access or from the file's fragmentation.

Solution 2

A: there is no problem. An available of almost 6 GB indicates plenty of headroom; the low free is the normal behavior of the page cache. The 2,100 MB in swap are historical: pages evicted long ago that nobody has needed since. si=0 and so=0 prove it —there is no swap traffic now— along with a memory pressure of 0.4%, that is, noise. Action: none, other than noting the value in the baseline. Emptying the swap "for tidiness" with swapoff -a && swapon -a is counterproductive: it forces pages nobody uses back into RAM.

B: a real and serious problem. An available of only 1,980 MB and, above all, si=1,840 and so=2,310 pages per second, that is on the order of 7 and 9 MB/s going in and out simultaneously: that is thrashing, the system is moving pages in both directions because the working set does not fit in RAM. The 63.7% pressure confirms that almost two thirds of the time is being lost. Actions, in order: identify the consumer with ps -eo pid,rss,comm --sort=-rss | head; check in dmesg whether the OOM killer has already acted; look at how RSS evolves to tell a leak from a legitimate load; as an immediate mitigation, apply or tune MemoryMax in the guilty service's unit (07-02) to bound the damage; and as a real fix, correct the leak or size the machine properly. Lowering vm.swappiness fixes nothing here: the problem is not that swap is being used, it is that there is not enough memory.

Solution 3

(a) "CPU > 85% for 1 minute" → "The p99 latency of /v1/readings exceeds 1 s for 10 minutes". The problem with the original is that it measures a resource, not harm: a well-sized machine ought to run high on CPU, and a service can be doing terribly with the CPU at 20% if the bottleneck is the disk. Besides, one minute is far too short and fires on any legitimate spike, such as the aggregator starting. The new version measures what the user suffers and its 10-minute window filters out transients. As a secondary, lower-priority alert, cpu some avg300 > 40% is defensible, because it measures lost time and not utilization.

(b) "Swap memory is in use" → "si+so > 0 sustained for 5 minutes", or better "memory full avg60 > 10%". Having pages in swap means nothing, as machine A in the previous exercise shows: the original alert would fire on perfectly healthy machines and would end up silenced. What hurts is the swap traffic, or more directly the lost time PSI measures, which is also comparable across different machines.

(c) "The meteo-api process is not running" → "meteo-api.service has gone 5 minutes without reaching the active state (or has entered failed)". The original is guaranteed noise: every legitimate restart, every deployment and every Restart=on-failure that works exactly as intended would generate a three-in-the-morning alert about something the system has already fixed by itself. The new one fires only when the automatic recovery has failed, which is the only case where a human is needed. It is exactly the stable state we achieved with StartLimitBurst in 07-02, and systemctl list-units --failed is the check that implements it.

The central message of this lesson is that method is worth more than tools. You can tell a symptom from a cause, quantify before touching anything, change one thing at a time and lean on a baseline; and you have two named frameworks: USE, to walk the resources with utilization, saturation and errors —remembering that saturation is the metric with no ceiling that almost nobody looks at—, and RED, to measure the service from outside with rate, errors and duration.

On top of that you have learned to read the numbers for real, and they almost always mean something other than they appear to. The load average includes processes in state D and is therefore not CPU utilization. %wa is idle time with I/O outstanding, so a low value does not rule out a disk problem. free confuses because the cache is not occupied memory and the good figure is available; swapping hurts when there is si/so, not when there is swpd. %util misleads on NVMe and the trustworthy metrics are await and aqu-sz. On a listening socket, Recv-Q is connections waiting for accept() and it gives away the application, not the network. And PSI measures lost time directly, in units comparable across machines, which is what turns an alert into something meaningful.

You have also seen the cost of observing: strace slows things down by 10 to 100 times because it intercepts, whereas perf samples at 99 Hz for 1-2% and eBPF aggregates inside the kernel; and you have moved up from the machine to the fleet, with exporters, time series, cardinality, percentiles, symptom-based alerts and error budgets derived from an SLO. In the lab you have provoked the three saturations and written down their signature, including the comparison that sums up the module: a 4 KB random read and a 1 MB sequential read give the same %util and incomparable user experiences.

You now have the three pieces: the interface for acting (07-01), the control over what runs and when (07-02) and the method for knowing what is going on. What remains is putting it all together under pressure, with incomplete data, at a bad hour and with the real uncertainty of not knowing the answer in advance.

That is the last lesson of the course: Final Case Study: Troubleshooting a Production Server. It is 03:12 and an alert has just come in.