You know what happened at three in the morning: fifteen db_timeouts with the 200 connections exhausted. What you do not know is why. And now Marta adds fuel to the fire: "Several of the sales team say the website is slow in the mornings, around nine. Is it something at our end?". "It is slow" is not a piece of data, it is a feeling; turning it into a number, finding the guilty resource and proving that the change worked is precisely the job of this lesson. You are going to learn a reproducible method instead of a collection of loose commands, to genuinely interpret what uptime, free or iostat say, and to establish a baseline for srv-tramontana without which no measurement means anything.

1. Monitoring, diagnosing and the USE method
2. CPU: load average, vmstat, mpstat and steal
3. Memory: what "used" means, swap and the OOM killer
4. Disk: iostat -xz, latency versus throughput
5. Network: sar -n DEV, ss -s and retransmissions
6. Overview tools and history with sar
7. Continuous metrics and the four golden signals
8. A procedure for "the server is slow"
9. Tuning within an administrator's reach
10. The srv-tramontana baseline
11. Tramontana case: "the website is slow in the mornings"

These are two different activities and they are constantly confused:

• Monitoring is knowing how the system is doing continuously and without anybody watching: metrics every fifteen seconds, stored, with alerts. It answers "is it all right?" before the customer calls.
• Diagnosing is finding out why it is going badly at a particular moment, with interactive tools and a hypothesis that is confirmed or discarded.

A human staring at top is not monitoring: it is manual, occasional diagnosis with no memory. You need both, and the first makes the second possible, because a measurement with nothing to compare it against says nothing: a load of 3.5 may be normal or catastrophic, and you only know which if you know yesterday's.

The USE method (from Brendan Gregg) gives the order in which to look. For each resource — CPU, memory, disk, network — three things are checked:

Its practical value is that it stops you wasting time: in five minutes you go through the four resources with three questions each and come out knowing which is the bottleneck.

The three numbers are the average of processes in the runnable (R) or uninterruptible-wait (D) state over 1, 5 and 15 minutes. Two rules change everything. First: you have to normalise by the number of cores; with nproc = 2, a load of 3.42 is 1.71 processes per core and the system is saturated, whereas the same 3.42 on a 16-core machine would be 21% occupancy, that is, nothing. Second: the D state counts, and that is the nuance almost nobody knows, because on Linux the load includes processes blocked waiting for disk; that is why you can see a load of 8 with the CPU at 5%.

The trend is informative too: 3.42 2.10 1.08 is a load that is rising (the problem is starting now); 1.08 2.10 3.42 is one that is already resolving itself.

The first line of vmstat is the average since boot: always ignore it and look from the second one onwards. In the example, wa at 56–68% with r at 4–5 and bo sky-high shouts that the bottleneck is the disk, not the CPU.

The st deserves a note: in a VM it is time the hypervisor gave to another machine, so an st of 20% means no optimisation of yours will fix anything, because the problem is on the host or with the provider.

And the breakdown by core and by process:

mpstat -P ALL reveals a very common case: the overall average looks reasonable but a single core is at 100% because the application does not parallelise. pidstat breaks the consumption down by process, and its %wait column — time spent waiting to be given CPU — is pure gold for spotting contention.

"Free: 208 Mi" is not a problem. It is exactly the opposite: free RAM is wasted RAM, and the kernel uses it as page cache so as not to read from disk again. The columns that matter:

used is process memory without cache or buffers; buff/cache is page cache and metadata, which the kernel releases instantly if it needs to; and available is the figure to look at: how much memory a new application can obtain without swapping. With 2.4 GiB available out of 3.8, srv-tramontana has plenty; the alarm would be available falling towards zero, not free.

Three figures that get confused every day: VSZ is the whole reserved address space — shared libraries and memory requested but never touched included — almost always enormous and irrelevant; RSS is the pages actually in RAM, useful but it counts each shared library in full in every process, so adding up the RSS values gives far more than the total RAM; and PSS divides each shared page between the processes using it, and is the only one that adds up properly (you see it with smem -k or in /proc/PID/smaps_rollup).

Having swap in use is not bad in itself: the kernel offloads pages nobody touches. What is serious is the constant traffic, which shows in si/so in vmstat. If those columns are sustainedly active, the system spends more time moving pages than working: that is thrashing, and it is felt as brutal slowness with the CPU almost idle.

When there is no memory and no swap left, the kernel picks a victim and kills it. Knowing it has acted is essential, because the symptom that reaches you is "the service restarted on its own":

It chooses by oom_score, which rewards killing the process that frees the most memory, weighted by oom_score_adj (from −1000 to 1000). It is queried in /proc/PID/oom_score and adjusted on the fly with sudo choom -p 1284 -n -500. Although in a systemd service the right thing is to declare it in the unit (OOMScoreAdjust=-500) and, above all, to set MemoryMax as you saw in 05-05: that way the OOM acts inside the service's cgroup and does not take the database down with it.

The critical distinction is latency versus throughput: a disk can move 200 MB/s quite happily with a sequential task (high throughput, low latency) and choke on 5 MB/s of small random writes (ridiculous throughput, terrible latency). What hurts the user is the latency, not the throughput.

And who is writing:

That last command is a preview of today's case: two hundred established connections to the database's port, exactly the max_connections in app.conf.

To see the traffic live, nload (per interface) or iftop (per conversation) are immediate.

And the TCP health indicator you have to know about, nstat -az | grep TcpRetransSegs: retransmissions are segments that had to be sent again because no acknowledgement arrived. A small percentage is normal; a rising rate indicates packet loss — a cable, a saturated link or an intermediate device dropping traffic. You already have the complete network diagnosis, with mtr and the layers, from 03-08.

The point most often overlooked: today's snapshot is useless without yesterday's. If Marta complains at 09:10 about something that happened at 09:00, top no longer says anything; sar does:

Enabling sysstat takes a minute and it is what turns "I think it was slow" into "at 09:04 the %iowait rose to 61%".

For more than one server, or to alert without anybody watching, today's standard is Prometheus (a time-series database that polls the targets), node_exporter (which exposes the system's metrics at /metrics), Grafana (dashboards) and Alertmanager (notifications). We are not setting it up here: it is the subject of Module 8. What you should take away is what to measure, the four golden signals:

And a rule about alerts: an alert that goes off every day stops being read. Alert on symptoms that hurt the user (p95 latency, error rate), not on every CPU spike. It is the same "silence if all is well" discipline as in your scripts.

This is the numbered procedure you can apply as it stands, in this order, without skipping steps:

1. Translate the complaint into a measurable number. Which operation, at what time, how long does it take now and how long before? Without this you cannot know whether you have fixed anything.
2. A 60-second snapshot. uptime, vmstat 1 5, free -h, iostat -xz 1 3, ss -s, systemctl --failed. With six commands you know whether the problem is CPU, memory, disk or network.
3. Compare with the baseline. Is it abnormal, or has it always been like that and what has changed is the expectation?
4. Identify the resource with USE and choose one, explicitly ruling out the others.
5. Locate the culprit within that resource with pidstat, iotop, ss or top -H: a name and a PID.
6. Formulate a falsifiable hypothesis. "The 04:20 backup saturates the disk and that is why the queries time out" can be confirmed or ruled out; "it is slow" cannot.
7. Review what changed: journalctl --since, /var/log/apt/history.log, the git log of your scripts, the HISTORY. 80% of new problems come from a recent change.
8. Measure before touching anything and save the figures in a dated file.
9. Change ONE single thing and document what and why.
10. Measure afterwards, under the same conditions, and compare. If it does not improve, revert before trying something else.
11. Write the result down in the HISTORY, even — especially — if the change did not help.

Steps 8, 9 and 10 are non-negotiable. Changing three parameters at once and seeing an improvement teaches you nothing: you do not know which one helped, and you will drag two useless settings around for ever.

Kernel parameters via sysctl are the subject of 07-03 and profiling with perf and eBPF is 07-02. What is in your hands today:

Open file limits. A server with many connections exhausts the per-process limit and rejects requests with Too many open files:

In a systemd service you do not touch /etc/security/limits.conf — which applies only to login sessions via PAM — but the unit:

The service's resource limits: MemoryMax and CPUQuota from 05-05, which as well as protecting are useful for measuring, because systemd-cgtop shows whether the service reaches its ceiling. And noatime in fstab (05-04), which avoids one write for every read and is noticeable on a server with many small files.

Shallow thread profiling: top -H -p 1284 breaks the consumption down thread by thread when a multithreaded process is loaded and you do not know which part. perf top would give the breakdown by function, but that is 07-02.

The application's own parameters, which usually pay off more than any system tuning: in Tramontana, max_connections and query_timeout in /etc/tramontana/app.conf.

A baseline is a snapshot of normal behaviour, taken when everything is fine, to compare against when it is not. It is kept alongside the reference one from Module 1:

Normal values measured on srv-tramontana on a Tuesday at 11:00, with the application running:

That table, and not intuition, is what turns a measurement into a diagnosis.

Step 1 — Translate the complaint. We ask and we measure: the availability search, which normally answers in 180 ms, takes between 4 and 9 seconds between 08:50 and 09:15. That is a number already.

Steps 2-3 — A snapshot and a comparison with the baseline. As the incident is recurring, we measure at 09:00 the next day:

A load of 3.42 with 2 vCPU (baseline: 0.15–0.40) and wa at 68% (baseline: 0–3%) with id at 8%. It is not CPU that is missing: it is disk. Memory and swap, normal. Resource identified: I/O.

Steps 4-5 — The culprit.

A w_await of 22.85 ms against the baseline's 0.8–2.5, %util at 97.6%, and the one doing the writing is the backup, which in theory is launched at 04:20: the one on the 17th took 1 h 42 min and the one on the 18th had still not finished at 09:04. The data set has grown and the backup overlaps with the peak hour.

Step 6 — A falsifiable hypothesis. "The backup saturates the disk's I/O until after 09:00; that lengthens the database queries, which hit the 30 s query_timeout and exhaust the 200 connections of max_connections, which is what causes the db_timeouts in the logs."

Checking the second half:

Exactly two hundred connections — the ceiling — and eleven errors in four minutes. Hypothesis confirmed: the same mechanism that caused the 03:00 db_timeouts in the previous lesson.

Steps 7-9 — What changed, measure and change ONE thing. The HISTORY says it: the backup volume was migrated on the 18th and the data set has grown. The root cause is the duration of the backup, not the connections. The minimal, reversible change:

Step 10 — Measure afterwards, at the same time and with the same commands.

Step 11 — Document. And the report for Marta, with the usual structure: "The morning slowness was caused by the backup, which had grown to take more than four hours and was still writing during peak hours; by saturating the disk, the queries timed out and the 200 database connections were exhausted. We have brought the backup forward to 02:30 and given it a 3-hour limit and lower I/O priority. Measured at 09:04: the search is back to 190 ms and there are no errors. What this protects: it stops the backup interfering with working hours. What it does not protect: if the data set keeps growing, we will hit this again; we need to move to incremental backups, and we are tackling that right now. Nor does it solve the fact that 200 connections is a tight limit if the number of users grows."

• Reading the load average without dividing by nproc. A 3.42 means nothing until you know how many cores there are.
• Panicking about "little free memory". The figure that matters is available; the cache is memory well spent.
• Trusting the first line of vmstat or iostat. It is the average since boot, not the current moment.
• Looking at %util on an SSD as if it were a mechanical disk. With parallel queues, an NVMe can show 100% and not be saturated. Look at await and aqu-sz.
• Diagnosing without a baseline. Without knowing what is normal, any number looks alarming or reassuring depending on your mood. And do not change several things at once: you will not know which one worked, and you will drag useless settings around for years.
• Having no history. Install sysstat on day one: by the time somebody asks about 09:00, it will be too late to enable it.
• Confusing a symptom with the cause. The 200 connections were the symptom; the backup was the cause. Raising max_connections would have hidden the problem.
• Tip: save every measurement in a dated file, alongside the HISTORY. In six months' time, that folder will be worth more than your memory.

1. Reading a snapshot. A server with 4 vCPU shows load average: 7.80, 7.20, 6.90, vmstat with us=6 sy=3 id=4 wa=87 st=0, r=1, b=6, and free -h with 5.1 GiB available out of 8. Which resource is saturated, which one is not, and what are the next two commands?
2. A service that dies in the small hours. tramontana.service shows up as restarted every night around 03:40 without anybody touching it. Give the commands to confirm whether it was the OOM killer and, if it was, two ways of stopping it taking the database down with it.
3. Proving an improvement. Design the exact protocol — commands, moments and success criterion — to demonstrate that enabling noatime on /srv/tramontana/backups shortens the nightly backup, in a way that would convince a sceptic.

1. The saturated resource is the disk. The proof: wa=87% with id=4% means the CPU is sitting still waiting for I/O, and b=6 is six processes blocked in state D, which are also what inflate the load to 7.80 despite there being only r=1 waiting for CPU. Normalised, 7.80 across 4 cores would be 1.95, but that number is misleading here precisely because it is made up of processes in D, not in R. What is not saturated: the CPU (us+sy = 9%) or the memory (5.1 GiB available out of 8, with no swap traffic). The next two commands:

2.

status=9/KILL without anybody having run a kill is the OOM killer's signature, and the kernel's message confirms it. Note the coincidence in timing with the nightly backup: the page cache the backup generates puts pressure on the available memory.

Two ways of protecting the database:

The first is the good one, because it contains the problem at its source: when Tramontana overruns its memory, Tramontana will die and systemd will bring it back up, without the kernel choosing a victim across the whole machine. The second is a defensive complement: it biases the choice, but it does not stop the machine running out of memory.

3. The protocol, which is the method from section 8 applied:

The success criterion, fixed before measuring: a reduction of more than 10% in the median duration with no increase in errors in journalctl -u tramontana-backup.service -p err. Three nights on each side because a single measurement cannot tell an improvement from a quiet night. And if the criterion is not met, you revert with the .bak and you write in the HISTORY that it did not help: a documented negative experiment saves somebody repeating it in a year's time.

You no longer depend on somebody "noticing" that the server is slow. You tell monitoring from diagnosing and you apply the USE method — utilisation, saturation, errors — to the four resources. You read the load average normalised by nproc knowing that it includes processes in state D, and you interpret vmstat 1 column by column: r and b, si/so, us/sy/id/wa and the st that gives away the hypervisor. You break the consumption down with mpstat -P ALL and pidstat.

In memory you know that "free" is a misleading figure and that the good one is available, you tell VSZ, RSS and PSS apart, you recognise thrashing in si/so, and you know how to prove the OOM killer acted with journalctl -k and how to stop it taking the neighbour with it using MemoryMax and OOMScoreAdjust. On disk you handle iostat -xz with %util, await and aqu-sz, you are clear about the difference between latency and throughput, and you locate the culprit with iotop and lsof. On network you use sar -n DEV, ss -s and retransmissions. You know htop, atop, glances and dstat, and — most importantly — you have enabled sysstat so as to have the history without which today's snapshot means nothing. You know what the four golden signals are and why an alert that goes off every day stops being read.

You have an eleven-step procedure for "the server is slow", with the three non-negotiable rules: measure before, change one single thing, measure after. You have a baseline for srv-tramontana with written thresholds. And you have solved the case: the morning slowness was the backup saturating the I/O until after 09:00, causing the db_timeouts and the exhaustion of the 200 connections; bringing it forward and capping it took the search back from 4–9 s to 190 ms, with the before-and-after figures written into the HISTORY.

But the report for Marta ended with a loose end that cannot wait: the full backup will keep growing, and the problem with it. And there is something far worse that nobody has checked yet: nobody has ever restored that backup. In Backup and Restore, the module's last lesson, you will see why backups do not exist, only tested restores: you will set an RPO and an RTO for Tramontana, you will apply the 3-2-1 rule and generational retention, you will solve consistency with the LVM snapshots from 05-04, you will use tar --listed-incremental, rsync --link-dest and a modern tool with deduplication and encryption, and you will write the step-by-step restore procedure for the three scenarios that can genuinely happen to you.