In the previous lesson you found a performance problem whose cause was a badly set configuration value: max_connections=200 against a max_connections=100. You found it by measuring, and you fixed it by checking the effect. This lesson does the same thing one level down: the values you are going to adjust are the kernel's.

And it starts with a warning, because kernel tuning is the area of systems administration with the most folklore per square metre. The Internet is full of lists of parameters "to optimise Linux" that get copied without being understood, that contradict the defaults for no reason, and that in the best case do nothing at all. There is a single rule that separates the work from the ritual:

You do not tune what you have not measured.

And the procedure that applies it, which is the same one you have already used twice — in the slow-website incident in 05-07 and in the latency one in 07-02:

1. Measure the current state and write it down. Without a baseline there is no "after".
2. Formulate a specific hypothesis: which parameter, why you think it matters, and what you expect to change.
3. Change one single thing. Two simultaneous changes make it impossible to attribute the result.
4. Measure again with the same method.
5. Document the change with its reason, or revert it. A parameter with no written justification is a parameter nobody will be able to remove in a year's time.

Ubuntu 24.04's defaults are reasonable for most workloads. Changing them without a measurement to back it up is, in all probability, making the system worse.

1. sysctl: the kernel's parameter interface
2. Virtual memory
3. The network and the TCP stack
4. File and process limits
5. The I/O scheduler
6. Transparent huge pages
7. Per-process limits and which one wins
8. Frequency governors and tuned
9. Kernel modules
10. Compiling the kernel: when it makes sense and when it does not
11. The Tramontana case: tuning the network stack with measurement

The kernel exposes its tunable parameters as files under /proc/sys/. It is Module 1's "everything is a file" applied to the kernel's own configuration: reading the parameter means reading a file, changing it means writing to it.

sysctl is the tool that does the same thing with a dotted syntax, where every dot is a slash in the path:

Almost thirteen hundred parameters, two thirds of which are network ones. The vast majority are never touched.

The three modes of operation, and which to use when:

That ephemeral -w is the tool for step 3 of the procedure: it lets you try a change, measure, and if things get worse a reboot — or rewriting the old value — is enough to undo it. You never put a parameter straight into /etc/sysctl.d/ without having tested it live first.

Persistence and its precedence:

The files are applied in alphabetical order, and the last one to write a parameter wins. Hence the convention of numbering them: 10- for whatever comes from the distribution, 60- to 90- for your own, and 99- for anything that must override everything. And an important note: /etc/sysctl.conf still works, but it is deprecated; use /etc/sysctl.d/ with separate files by purpose.

Notice that you already have 60-hardening.conf, from 06-06, with the security parameters. The performance ones go in a separate file, and that separation is not cosmetic: when somebody has to review the security posture, or when a performance setting has to be reverted, only one file is touched and what can be removed does not get mixed up with what cannot.

The virtual memory subsystem is where settings have the greatest impact and where there is the most mythology. The parameters that matter:

The most misunderstood of the lot. It is not "how much swap to use", nor a percentage of memory. It is the kernel's relative preference between two ways of freeing memory when it needs to: discarding file cache pages, or moving anonymous pages (process memory) to swap.

And the fact to internalise before touching it: if the system is not using swap, swappiness does nothing. It is a parameter that only manifests itself under memory pressure. Before changing it, measure whether there is any pressure:

Zero swap-ins and zero swap-outs since boot. On srv-tramontana, with 3.8 GB and 2.4 GB available, swappiness is irrelevant today. Lowering it would improve nothing.

Is it worth changing, then? Yes, but for a different reason than average performance: as insurance against degradation under pressure. If one day the application grows and pressure starts to appear, with swappiness=10 the kernel will prefer to discard cache rather than send the process's active pages to swap, which avoids the scenario of hundreds of milliseconds of latency. It is a defensible decision, and that is how it has to be documented: not "this speeds up the server", but "this limits the damage if memory ever runs short".

These do have a measurable effect, and they are the cause of a very characteristic latency pattern.

When a process writes to a file, the data goes to the page cache first and is marked as dirty; the kernel flushes it to disk later. These two parameters control when:

The pattern they produce: fast writes while there is headroom, and then suddenly a long pause when dirty_ratio is reached and the process is left blocked. With 3.8 GB of RAM, 20% is about 760 MB of dirty data that may have to go to disk all at once.

That is exactly the profile of the 05-07 incident: the backup writing in bursts, with a w_await of 22.85 ms and the application blocked. ionice mitigated the symptom; these parameters attack the mechanism.

Smaller values produce more frequent, smaller flushes, with more uniform latency:

And the honest warning: this does not make the system "faster" in total throughput — sometimes it makes it slightly slower — but more predictable. For an interactive service, uniform latency is worth more than peak throughput. For an isolated backup job, the opposite. It is a trade-off, not an improvement.

vm.overcommit_memory=2 deserves a warning: it looks prudent ("do not promise memory you do not have") and in practice it makes perfectly normal applications fail on allocation, because many reserve far more than they use. It is not touched without a very specific reason.

This is where srv-tramontana has real headroom, and where the 07-02 incident left a clue unexplored.

When a TCP connection arrives, it passes through two queues before the application accepts it:

And the crucial detail that makes this parameter get tuned wrongly half the time: somaxconn is a ceiling, not an effective value. The real queue is the minimum of somaxconn and the backlog parameter the application passes to the listen() call. Raising somaxconn to 65535 achieves nothing if the application asks for 128.

The measurement that says whether the queue is filling up:

That Send-Q 128 on port 8080 is the valuable piece of information: the application asks for a backlog of 128, so the somaxconn of 4096 does not affect it at all. And ListenOverflows at zero means no connection has ever been dropped for a full queue. Conclusion: raising somaxconn on this machine would do nothing, and whoever put it in their sysctl.d would be adding folklore.

This is the procedure working: the reasonable hypothesis ("the connection queues are the bottleneck") is refuted by the measurement before anything is touched. Refuting hypotheses is as valuable as confirming them, and considerably cheaper.

Here there really is something the 07-02 incident left in plain view. When the application was opening 300 connections every ten seconds, each one consumed an ephemeral port that stayed in the TIME_WAIT state for 60 seconds:

Seventy-one sockets in TIME_WAIT. With 28,231 ports available that is not a problem, but during the incident, at 30 connections a second, some 1,800 built up — and an application opening ten times more would exhaust them.

TIME_WAIT is not a defect: it guarantees that delayed packets from a closed connection do not get confused with a new one reusing the same pair of ports. The possible adjustments:

That last one is the perfect example of folklore: tcp_tw_recycle appears in hundreds of guides "to optimise TCP", it broke connections from NATed networks subtly and intermittently, and it was removed from the kernel years ago. If you find a guide recommending it, you know its author has not tested it this decade.

On Ubuntu 24.04, tcp_tw_reuse already comes set to 2 (enabled only for loopback and link-local addresses), which is a prudent value. The real fix for the 07-02 problem was not a sysctl at all: it was fixing the connection pool, which is what you did.

The three values of tcp_rmem are minimum, default and maximum, and the kernel adjusts automatically within that range. Widening the maximum matters when the bandwidth × latency product is large: a 1 Gbit/s link with a 100 ms round trip needs about 12 MB of window to saturate, and with a 6 MB maximum it stops halfway.

On a local network with 0.4 ms of latency, like Tramontana's, this is completely irrelevant. It is a setting for long-distance transfers, and putting it in "just in case" only consumes memory.

Congestion control decides how fast TCP sends when it detects problems on the network. The classic algorithm, CUBIC, interprets packet loss as a congestion signal. BBR, developed by Google, instead models the path's real bandwidth and latency.

The practical difference is large on links with loss or with large buffers (the bufferbloat problem), which is the case for nearly any connection over the Internet:

From 77 to 185 Mbit/s on the same link, and with lower latency under load. default_qdisc=fq is not optional: BBR needs fair queue queueing to work as designed, and without it the result is worse.

With the measurement done, the change is made persistent with its reason written down:

Notice the comment: what was measured, with what, the result, the date and how to revert it. That is the difference between a tuning decision and a copied line.

fs.file-max on modern kernels is practically unlimited, so raising it — another classic of the guides — does nothing. The real limit reached in practice is the per-process one, covered in the next section.

fs.inotify.max_user_watches does genuinely run out, and with a baffling error. Every inotify watch consumes an entry, and development tools, file synchronisers and logrotate use them heavily:

That ENOSPC which is not a shortage of disk is one of the most frequent confusions in Linux, and it deserves a place in the runbook.

The scheduler decides in what order requests to the disk are served. Ubuntu 24.04 uses the multi-queue infrastructure (blk-mq):

The none for NVMe rule has a specific reason behind it that you need to understand: a scheduler reorders requests to minimise the head movement of a mechanical disk. An NVMe has no head, serves tens of thousands of operations a second and has its own queues in hardware. Reordering only adds latency and CPU consumption. On a mechanical disk, by contrast, reordering can multiply the throughput.

Since /sys is not persistent, the correct way to fix it is a udev rule that decides according to the device type:

And two other queue parameters that sometimes matter:

read_ahead_kb is how much the kernel reads ahead. Raising it (to 512 or more) helps with large sequential reads — a backup, a database dump; lowering it helps with small random accesses, because it avoids reading data that will not be used. It is exactly the kind of trade-off to measure before touching.

An observation about Tramontana's stack: the backup volume is LUKS on LVM on sda, and each layer presents its own block device with its own queue. The scheduler that matters is the physical device's (sda); the dm-* devices pass requests downwards. It is the same stack you had to break apart in the first exercise of 07-02.

The processor translates virtual addresses to physical ones in 4 KB pages, using a cache called the TLB. With a lot of memory, the TLB misses often and each miss costs extra memory accesses. Huge pages of 2 MB drastically reduce the number of entries needed.

THP (Transparent Huge Pages) does that automatically. And it is the parameter every database asks you to disable:

Why databases reject it, which is the interesting part: to hand out a 2 MB page, the kernel needs 2 MB that are physically contiguous. When memory is fragmented, it has to compact it, and that compaction happens synchronously in the context of the process asking for memory. The result is unpredictable pauses of tens or hundreds of milliseconds in exactly the process that tolerates them least. PostgreSQL, MongoDB, Redis, Oracle and Elasticsearch all document the recommendation of madvise or never.

madvise is the best compromise: the processes that benefit ask for it explicitly, and the rest do not pay the cost.

Since /sys does not persist, you fix it with a systemd unit — applying 05-05 — which is cleaner than an rc.local:

The Before=postgresql.service matters: the change has to be in place before the database allocates its shared memory.

A kernel limit is global; the ones reached in practice are the per-process ones, and here there are three mechanisms that tread on each other. Knowing which one wins saves hours of bewilderment.

And the answer to "which one wins", which is the reason for this section: for a systemd service, the unit wins, and limits.conf does not apply at all. systemd starts services directly, without going through PAM. Putting svc-tramontana hard nofile 16384 in limits.conf and expecting it to affect tramontana.service is a classic mistake that gives no warning whatsoever: it simply does not work.

With MemoryMax=512M and TasksMax=64 already set back in 05-05, the open file limit is adjusted the same way, by drop-in:

214 descriptors in use with a limit of 8192: plenty of margin without being absurd. That is the criterion — a limit is sized from measured usage, not from a round number somebody copied.

On a virtual machine the frequency management is done by the host, so this section does not apply to srv-tramontana. On physical hardware:

The real use case for performance is tail latency: raising the frequency takes microseconds, and in a service with strict latency targets those microseconds show up in the 99th percentile. At the price of considerably more electricity.

tuned is the sensible way to apply coherent sets of settings rather than isolated parameters:

That last command is the important piece of advice: tuned is useful because it applies tested, consistent sets of settings, but reading what a profile does before activating it is compulsory. A profile can change twenty parameters, and if one of them conflicts with your sysctl.d, the outcome depends on the order of application and is hard to debug.

A module is a piece of kernel that is loaded and unloaded on the fly: drivers, filesystems, protocols.

The persistent configuration:

Two important details. The first: install <module> /bin/true is more effective than blacklist <module>, because blacklist only prevents automatic loading and does not stop another module loading it as a dependency. The second: after touching modprobe.d, you have to regenerate the initramfs, for the reason you learned in 07-01 — the initramfs carries its own copy of that configuration.

This list of blocked modules is in fact a CIS Benchmarks recommendation that was left outstanding on the 06-06 checklist. It is a good moment to close it.

DKMS (Dynamic Kernel Module Support) solves the problem of third-party modules: a module compiled for kernel 6.8.0-41 does not work on 6.8.0-45. DKMS recompiles it automatically at every kernel update:

With Secure Boot active, a module recompiled by DKMS needs a signature, and that is where the mmx64.efi (MokManager) you saw in 07-01 comes in. It is the usual cause of a third-party driver ceasing to work after a kernel update on a machine with Secure Boot.

Compiling the kernel is a rite of passage in learning Linux, and it is worth being honest about its usefulness on a production server: hardly ever.

And the cost that gets underestimated: a hand-compiled kernel falls outside the package system. It receives no automatic security updates — goodbye to the unattended-upgrades of 05-03 — it is not signed for Secure Boot, and every kernel CVE demands recompiling by hand. On a machine handling personal data, that is a compliance problem, not just an inconvenience.

The packaged variants cover almost everything:

The procedure, for the lab:

make olddefconfig starting from /boot/config-$(uname -r) is what avoids the beginner's mistake: make defconfig generates a generic configuration that probably does not include the driver for your disk controller, and the result is the (initramfs) prompt from 07-01.

The complete exercise, applying the five-step procedure. Starting point: the 05-07 baseline, the incident of the db_timeouts with active_connections=200, and the connection pool adjustment from 07-02.

Three reasonable hypotheses drawn from the history, and what the measurement says about each:

Two out of three refuted by measurement, and that is the normal result. If you had copied a guide on "network optimisation for Linux", you would have set somaxconn = 65535 — which does nothing here — and probably tcp_tw_recycle, which no longer exists.

The final section — "WE DO NOT TOUCH THESE, and here is why" — is the part of the file with the most long-term value. It documents the refuted hypotheses, and that stops somebody in six months' time — you included — adding somaxconn = 65535 because they saw it in a guide. A setting discarded with its reason written down is knowledge; a setting that is simply absent is only an omission.

• Copying sysctl lists off the Internet. Most are ten years old, many contradict defaults that are already correct, and some recommend parameters that no longer exist (tcp_tw_recycle is the unmistakable sign of an untested guide).
• Changing several things at once. If it improves, you do not know which; if it gets worse, you do not either. One change, one measurement.
• Putting a parameter in /etc/sysctl.d/ without testing it with -w. The ephemeral change is reversible with a reboot; the persistent one can leave you with a system that boots badly.
• Believing swappiness reduces memory usage. It only acts when there is pressure and the system resorts to swap. With no swap in use, it does nothing.
• Raising somaxconn without looking at the application's backlog. The effective limit is the minimum of the two. ss -ltn shows the real one in Send-Q.
• Trusting limits.conf for a systemd service. It does not apply: systemd does not go through PAM. You use LimitNOFILE= in the unit, and verify in /proc/<pid>/limits.
• Leaving THP at always with a database. It causes unpredictable synchronous compaction pauses. madvise is the right compromise, and it has to be applied before the database starts.
• Setting mq-deadline on an NVMe. Reordering requests makes no sense with no head to move: it only adds latency. none for NVMe.
• Forgetting update-initramfs after touching /etc/modprobe.d/. The initramfs carries its own copy. It is the 07-01 lesson.
• Compiling the kernel in production. It falls outside the package system: no automatic security updates, no signature for Secure Boot, and every CVE handled by hand. Try linux-image-virtual or the HWE kernel first.
• Tuning with no baseline. With no recorded "before", the "after" means nothing, and the feeling of improvement is notoriously unreliable.
• A tip on method. Each parameter's comment should answer four questions: what you measured, with which command, what the result was, and how to revert it. And document what you decided not to touch as well: that is what stops the folklore getting back in.

Luis sends you this list "to optimise the server", taken from a blog:

Evaluate each line: what it would do, whether it makes sense on srv-tramontana, and with which command you would check it. Write the reply you would send him.

After the 07-02 adjustment, the application keeps a pool of 80 connections to PostgreSQL. Check whether the file descriptor limits are adequate at the three levels involved (the application's service, PostgreSQL's, and the global one), and adjust whatever is needed, with its measurement.

/srv/tramontana/backups is LUKS on LVM on sda, and you want to know whether the I/O scheduler affects the duration of the nightly backup. Design the experiment: what you would measure, how you would isolate the variable, and why the result might not be conclusive.

Solution 1

Re: kernel optimisation list

Thanks, Luis. I have gone through it line by line against the server's real state. Summary: one makes sense, one is counterproductive, one does not exist, and four would do nothing. Here is the detail with the commands, so you can check it yourself.

---

1. net.core.somaxconn = 65535 — No effect

somaxconn is a ceiling. The real queue is the minimum of this value and the backlog the application asks for in listen().

$ ss -ltn | grep 8080
LISTEN  0  128  127.0.0.1:8080  0.0.0.0:*
$ nstat -az | grep ListenOverflows
TcpExtListenOverflows  0  0.0

Send-Q = 128: the application asks for 128, so the current value (4096) is already irrelevant, and 65535 would be just as much so. And ListenOverflows = 0 means no connection has ever been dropped for a full queue. For this parameter to do anything we would have to change the backlog in the application's code, and there is no evidence at all that it is needed.

2. net.ipv4.tcp_tw_recycle = 1 — It does not exist

$ sysctl net.ipv4.tcp_tw_recycle
sysctl: cannot stat /proc/sys/net/ipv4/tcp_tw_recycle: No such file or directory

It was removed from the kernel in version 4.12 (2017) because it broke connections from NATed networks intermittently and in a way that was very hard to diagnose. Its appearing in the guide tells us something useful: the author has not tested it this decade, so the rest is worth being sceptical about. If the aim was recycling TIME_WAIT, the current parameter is tcp_tw_reuse, and it is already at 2, which is the prudent default on Ubuntu 24.04.

3. net.ipv4.tcp_max_syn_backlog = 65535 — No measurable effect

$ nstat -az | grep -iE 'TCPReqQFullDrop|ListenDrops'
TcpExtListenDrops  0  0.0

This is the queue of half-open connections, and its real use is surviving a SYN flood. Zero drops with the current value of 1024. Besides, against that attack we already have tcp_syncookies enabled since 06-03, which is the correct defence. Raising it would reserve kernel memory for no benefit.

4. fs.file-max = 2097152 — No effect (and it would be a reduction)

$ sysctl fs.file-max
fs.file-max = 9223372036854775807
$ sysctl fs.file-nr
fs.file-nr  2848  0  9223372036854775807

On modern kernels it is already practically unlimited: setting 2,097,152 would be lowering it. And we use 2,848. The limit that is actually reached in practice is the per-process one, which on a systemd service is set with LimitNOFILE= in the unit — limits.conf does not apply to services, a detail that confuses a lot of people.

5. vm.swappiness = 0 — Counterproductive

This is the one that worries me. swappiness = 0 does not mean "do not use swap", it means "use swap only to avoid the OOM killer". The real effect is that the kernel exhausts the cache completely before considering swap, and in a memory spike it can invoke the OOM killer prematurely — killing the application's process or PostgreSQL's.

$ grep -E 'pswpin|pswpout' /proc/vmstat
pswpin 0
pswpout 0
$ free -h | awk 'NR==2 {print "available:", $7}'
available: 2.4Gi

Today it makes no difference because there is no swap activity, but the day there is, 0 is worse than the default. I have set 10, which prefers discarding cache without giving up swap as a safety net, and I have documented it as insurance against degradation, not as a performance improvement.

6 and 7. rmem_max / wmem_max = 128 MB — No effect, and it consumes memory

Large buffers matter when the bandwidth × latency product is large.

$ ping -c3 10.0.2.15 | tail -1
rtt min/avg/max/mdev = 0.312/0.398/0.482/0.071 ms

With a 0.4 ms round trip on a local network, the window needed to saturate 1 Gbit/s is about 50 KB. The current maximum (208 KB of rmem_max, and up to 6 MB in tcp_rmem) is ample. Reserving 128 MB per socket on a machine with 3.8 GB of RAM is a real risk if many connections are opened.

---

What I have done, and why

Following the same criterion — measure first — the only network change that turned out to be justified is BBR congestion control, which was not on your list:

# cubic (the default)
$ iperf3 -c 198.51.100.20 -t 20 -R | tail -2
[  5]   0.00-20.00  sec   184 MBytes  77.2 Mbits/sec  receiver
# bbr + fq
$ iperf3 -c 198.51.100.20 -t 20 -R | tail -2
[  5]   0.00-20.00  sec   441 MBytes  185 Mbits/sec  receiver

From 77 to 185 Mbit/s towards remote clients. That is a setting with a measured effect, and it is in /etc/sysctl.d/70-performance.conf with the measurement, the date and the way to revert it.

What I propose for next time. Before applying a parameter, three questions: which counter tells me this resource is the bottleneck? have I tested it with sysctl -w and measured before and after? is the reason written down? If any answer is no, the parameter does not go in. I have added a "WE DO NOT TOUCH THESE, and here is why" section to the file with these seven lines and their reasoning, precisely so that we do not have to discuss it again in six months.

Solution 2

The three levels, measured from the inside out.

Real usage 214 against a limit of 8192: comfortable and correct. The breakdown confirms the number makes sense:

80 sockets = the PostgreSQL connection pool, exactly the max_connections=80 value set in 07-02. The descriptors are where they should be.

The soft limit is 1024, and max_files_per_process is 1000. And here is the calculation that reveals the risk: each PostgreSQL backend process opens descriptors for the table and index files it touches. With 100 possible connections and up to 1000 files per process, the main process's limit of 1024 is tight.

It is not being reached today, but the margin is slim and the failure, when it comes, shows up as intermittent connection errors under load — hard to diagnose.

2,848 descriptors across the whole system against a practically infinite limit: there is nothing to adjust, and this confirms what Luis was told in the previous exercise about fs.file-max.

There is a judgement call here: the global default could be raised, but it is preferable not to. A high default limit hides descriptor leaks: a service that does not close what it opens fails early and visibly with a reasonable limit, and silently consumes kernel memory for months with a high one. Better explicit limits, sized per service.

The final verification:

Zero EMFILE/ENFILE errors, which are the ones that would appear if descriptors ran out. And the lesson on method: the problem was in PostgreSQL, not in the application, even though the earlier symptom manifested in the application. Just as in 07-02, the cause was on the other side of the client-server relationship. It is worth adding the rule to the runbook: when adjusting a limit on one side, check the one on the other.

Solution 3

What I would measure, and why those metrics. The backup's total duration is the variable of interest, but it is far too coarse to attribute to the scheduler. You have to measure at three levels:

How I would isolate the variable. This is the core of the exercise, and there are four sources of contamination to neutralise:

And an important design decision: the experiment is done on srv-tramontana-test, not in production. It is the VM you will create in the next lesson, and this is exactly the use case that justifies it: changing the I/O scheduler live on the real server, with drop_caches and without ionice, is asking for a service degradation.

Why the result might not be conclusive. Five reasons, and they are what makes this a good exercise:

1. The scheduler that matters is the physical device's, not the stack's. /srv/tramontana/backups is dm-1 (LUKS) on lv-backups (LVM) on sda. The dm-* devices have no scheduler of their own: they pass requests downwards. You check it:

   $ cat /sys/block/dm-1/queue/scheduler
   none
   

   A fixed none, with no alternatives in square brackets, indicates that device does no scheduling. So the experiment only makes sense on sda.

2. We are on a virtual machine. The guest's scheduler operates on a disk that is really a file on the host, which has its own scheduler and its own cache. The guest's decisions can be completely overridden by the host. This is what will probably make the result inconclusive, and it is the reason virtual-guest is the active tuned profile.

3. The bottleneck could be the encryption, not the I/O. If perf stat gives CPUs utilized close to 1.00 and offcputime points at AES functions, the process is CPU-bound and the I/O scheduler is irrelevant by definition. This check has to be done before the experiment: if the bottleneck is the encryption, the experiment is moot.

4. The backup is mostly sequential writing, and there the differences between schedulers are small. Where they really diverge is on a mixture of random reads and writes with several processes competing.

5. restic deduplicates and compresses, so the relationship between data read and blocks written is not fixed. Even with identical source data, the I/O load can vary between runs.

The design's conclusion. The experiment should be done, but with the right expectation: the most likely hypothesis is that the scheduler has no measurable influence on this machine, and step 3 — measuring whether the bottleneck is CPU or I/O — will probably demonstrate it before you even start. That is not a failure: it is refuting a hypothesis for 30 minutes' work, and it avoids adding a udev rule that does nothing.

And if step 3 shows the bottleneck is the encryption, the correct line of investigation is a different one: check whether the CPU exposes AES-NI to the guest (grep -m1 aes /proc/cpuinfo) and, if not, enable host-passthrough in the VM's definition. That can make encryption five or ten times faster, and it is a virtualisation change, not a kernel one. Precisely the territory of the next lesson.

You know how to tune the kernel, and — more importantly — you know when not to tune it. You have the map of the parameters that genuinely matter: virtual memory with swappiness and the dirty page thresholds, the network stack with its two connection queues and congestion control, the file limits with the limits.conf trap that does not apply to systemd services, the I/O scheduler with the none-for-NVMe rule and the reason behind it, and the transparent huge pages that every database asks to be set to madvise. You have seen modules, modprobe.d with install ... /bin/true — closing an outstanding row on the 06-06 checklist along the way — DKMS, and an honest answer on compiling the kernel: hardly ever in production, because it takes the machine out of the package system and with it out of security updates.

But what you take away from this lesson is not a list of parameters. It is the procedure: measure, formulate a hypothesis, change one thing with sysctl -w, measure again, and document with the measurement and the date or revert. Applying it, of three reasonable hypotheses about srv-tramontana's network two were refuted — somaxconn plays no part because the application asks for backlog=128, and swappiness does nothing with no swap activity — and only one, BBR, turned out to have a measurable effect: from 77 to 185 Mbit/s. That ratio is the normal one, and the "WE DO NOT TOUCH THESE, and here is why" section of your 70-performance.conf is what will stop the folklore getting back in six months from now.

Notice where the last exercise has left you. To test the I/O scheduler without degrading the service you needed a machine where you could drop the cache, remove the ionice and stop the application without anybody noticing. To measure whether LUKS encryption runs in software you needed to change the VM's CPU configuration. And the 3.3.0 deployment that would not start and forced a rollback in Module 4 was never diagnosed, because there was nowhere to reproduce it. Everything points the same way: a test environment is missing, and the course has been feeling its absence for four modules.

In lesson 07-04: Virtualization with Linux you build one. You will understand what virtualising really is — the three models, the exact role of KVM as the module that turns Linux into a hypervisor, and its relationship with QEMU and libvirt — you will manage machines with virsh and virt-install, provision them unattended with cloud-init, choose between raw and qcow2 knowing why, take snapshots while remembering that a snapshot is not a backup, and connect the network by NAT or by bridge as appropriate — including the explanation of that virbr0 with no carrier that cost 35 seconds of boot time in 07-01. At the end you will have srv-tramontana-test, cloned from the real server: the place to test this lesson's kernel changes, reproduce the deployment that failed, and rehearse the boot recovery of 07-01 without risking anything. And along the way you will see the virtualisation stack from the inside, which is what makes the following lesson — containers — comprehensible by contrast rather than by analogy.