The course has spent four modules missing the same thing. In Module 4, release 3.3.0 would not start and deploy.sh performed an automatic rollback — correctly — but nobody ever got to diagnose why it would not start, because there was nowhere to reproduce it. In 07-01 you practised boot recovery by breaking the real server's fstab, with a snapshot taken first and your fingers crossed. In 07-03, the last exercise called for dropping the page cache, removing the ionice and stopping the application in order to measure the I/O scheduler: three things you do not do in production. And checking whether the CPU exposes AES-NI to the guest was left outstanding, which is a change to the virtual machine's configuration.

All of it points to the same gap: a test environment is missing. This lesson builds one, and along the way it teaches you the virtualisation stack from the inside — which is what will make the next lesson, containers, comprehensible by contrast rather than by the wrong analogy.

1. What virtualising is: the three models
2. Type 1 and type 2 hypervisors
3. Virtualisation versus containers
4. KVM, QEMU and libvirt: who does what
5. Installation and checking for support
6. virsh: managing machines from the command line
7. Creating a machine with virt-install
8. Unattended provisioning with cloud-init
9. Storage: formats, pools and snapshots
10. Networking: NAT, isolated and bridged
11. Performance: virtio and CPU configuration
12. Cloning and creating srv-tramontana-test

Virtualising is running a complete operating system inside another, making it believe it has hardware of its own. The underlying technical problem: certain CPU instructions are privileged and only the kernel may execute them. If the guest system believes itself to be the kernel and executes one of them, it has to be intercepted and something done with it. The three ways of resolving that define the three models:

Emulation still has its uses: it is what lets you run ARM on x86 (qemu-system-aarch64) or a system from 1985. For virtualising Linux on Linux on the same architecture, it is absurdly slow.

Paravirtualisation was the dominant technique before 2006 (Xen popularised it) and it demanded a modified kernel. Today it survives in something far more important than it looks: the paravirtualised drivers, virtio, which you will see later on and which are the difference between a slow VM and a fast one.

Hardware-assisted virtualisation is what srv-tramontana uses and what is used everywhere today. Intel calls it VT-x and AMD calls it AMD-V; both add a privilege level below the kernel where the hypervisor lives.

And KVM defies that classification, which is the interesting part: it is a Linux kernel module, so technically there is an operating system underneath (type 2), but that operating system becomes the hypervisor when you load it, with direct access to the virtualisation extensions (type 1). The honest answer is that it is a hybrid, and in practice it performs like a type 1.

Your laptop is running VirtualBox, which is type 2. srv-tramontana is a VM inside it. And in this lesson you are going to create a VM inside srv-tramontana, which is called nested virtualisation and requires VirtualBox to expose the CPU extensions to the guest. We will come back to that in the installation section.

This comparison is the bridge to the next lesson, and it is worth seeing before you touch containers so that you do not pick up the wrong analogy:

The structural difference is in a single row of the diagram: every VM has its own kernel; containers share the host's. Every other difference follows from that:

And the two consequences to hold on to:

• For security isolation, the VM is superior. A container escape is a failure in the shared kernel and gives access to the host. A VM escape requires breaching the hypervisor, which is a far smaller surface. That is why cloud providers run different customers' workloads in separate VMs, not in containers on the same kernel.
• For density and speed, the container wins by orders of magnitude. Starting a hundred containers is a matter of seconds; a hundred VMs are a hundred kernels.

They are not alternatives: they combine. What is usual today is running containers inside virtual machines, taking the VM's isolation between tenants and the container's density within each one.

Three pieces that get confused constantly. Each solves a different problem:

The division of labour between the first two is the key: KVM handles the CPU and the memory, which is where performance matters; QEMU emulates the devices, which are accessed far less. Without KVM, QEMU would emulate the CPU as well and you would have the slow model from the first section. The combination is usually written "QEMU/KVM".

And libvirt contributes something that is not obvious until you manage more than two machines: a stable abstraction layer. QEMU's command line is enormously long and changes between versions; libvirt defines the machine in XML, and virsh always speaks the same way. It also manages virtual networks, storage pools and permissions, and it is the API used by virt-manager, Vagrant, OpenStack and Terraform.

First, check that hardware support is there. And here the complication of nested virtualisation shows up:

srv-tramontana is a VirtualBox VM, and by default VirtualBox does not expose the virtualisation extensions to the guest. You have to enable it from the host, with the machine powered off:

On VMware the equivalent option is Virtualize Intel VT-x/EPT, and on a physical server it is enough to enable VT-x/AMD-V in the UEFI. After powering back on:

If kvm-ok still fails, virtualisation will work by pure emulation: usable for trying out the mechanics of this lesson, unusable for real work.

Note qemu:///system versus qemu:///session: the first is system machines, managed by the daemon as root, with access to the shared networks and pools. The second is user machines, unprivileged, with limited networking. For a server, always system, and it is worth fixing it so you do not have to type it:

And an important check that closes the circle with 07-01: installing libvirt brings up virbr0, the default NAT network's interface, and it is exactly the one that cost 35 seconds of boot time because systemd-networkd-wait-online was waiting for it to get a carrier:

It is still at 12 seconds because in the 07-01 exercise you limited systemd-networkd-wait-online to enp0s3 with a drop-in. Without that fix, installing libvirt would have added 35 seconds to the boot all over again, and probably nobody would have connected the two things.

virsh is the main tool. Its essential operations:

The distinction between shutdown and destroy deserves emphasis because the name is misleading: destroy deletes nothing, it is the equivalent of pulling the cable. It does not destroy the definition or the disks, but it can leave the guest's filesystem inconsistent. It is the same logic as shutdown versus pulling the cable from 01-05, and the correct order is always to try shutdown first and wait:

It is exactly the SIGTERM → wait → verify → SIGKILL pattern you learned in 03-06, applied to whole machines.

And virsh console is the tool that makes everything in 07-01 practicable: it gives access to the guest's serial console, so you can see the GRUB menu, enter emergency mode and repair a broken fstab with no need for a graphical interface. You exit with Ctrl+].

virt-install is the non-interactive way of defining and starting a machine. The complete example, commented line by line:

The last two go together and they are the pair that makes virsh console work: without console=ttyS0 on the kernel command line, the guest would write to a virtual screen nobody is looking at, and you would see a blank console. It is a baffling and very common failure.

An interactive installation from the Ubuntu installer takes twenty minutes and requires answering questions. For a test environment you want to be able to recreate, that is no use. The alternative is the next section.

The distributions publish cloud images: pre-installed disks, with cloud-init inside, that configure themselves on first boot from some data you hand them.

Note the difference between virtual size and disk size: 3.5 GiB declared, 588 MiB occupied. That is qcow2's thin provisioning, explained in the next section.

The configuration data is delivered in two YAML files:

The two files are packed into a tiny ISO image that the machine reads as though it were a CD:

And now the machine is created, with one important detail: the downloaded image is not used directly, a new disk is created that uses it as a backing file:

Under two minutes, without a single question, and with the production configuration already applied. And — this is what makes the method valuable — the user-data is a text file that gets versioned in git: the test machine is reproducible. It is the same infrastructure-as-code idea you will see with Ansible in 07-06, in its simplest form.

Look at the difference between ls and du for demo.raw: ls says 10 GB, du says 0. It is a sparse file, so raw also does thin provisioning on filesystems that support it. The real difference between the formats lies elsewhere:

The practical decision: qcow2 unless you measure that the 2-5% of performance matters. The snapshots and the backing files are worth far more than that margin in a test environment, and for production there are better alternatives than either (a direct LVM volume, which is what many hypervisors use).

The backing file is the piece that makes a lab efficient: the base image is shared and each VM only stores its differences. Ten test machines take up the base plus ten small difference files, not ten complete systems. With one rule that has to be respected: if you modify the base image, every disk backed by it is corrupted. The base is read-only, in practice.

libvirt organises storage into pools, which abstract away where the disks live:

A pool can be a directory, an LVM volume group, an iSCSI device, NFS or Ceph. The logical type over LVM is interesting here: vg-data has existed since 05-04 and using logical volumes directly as disks avoids the file layer.

And the warning, carrying the same weight as the one in 05-04 about RAID:

A snapshot is not a backup. It lives in the same file (or beside it), on the same disk, on the same machine. If the disk fails, if the file is corrupted or if somebody deletes the directory, the snapshot and the original go together. It meets none of the three requirements of the 3-2-1 rule from 05-08.

What a snapshot is: a fast point of return for a risky operation. Exactly what you need before trying a kernel change. And it has two costs you need to know about: every active snapshot degrades write performance, because each modified block requires copying the original first; and accumulating dozens of chained snapshots can make the disk unmanageable.

Snapshots taken with the machine running (--live) also capture the memory and are far more delicate: if the application has data half written, the restored state can be inconsistent. The --quiesce option asks the guest agent to flush the filesystems first, and it requires qemu-guest-agent installed inside. For a test environment, shutting the machine down and taking the snapshot cold is simpler and more reliable.

The three modes, and when to use each:

Under NAT, libvirt sets up a bridge (virbr0), a DHCP and DNS server (dnsmasq) and the address translation rules. The VM reaches the Internet with the host's IP, and nobody on the network can initiate a connection towards it. For a test environment that is the right thing: isolation by default with nothing to configure.

And this is a good place to close the matter of virbr0 and the boot, because the complete mechanism is now visible: the interface exists from the moment libvirt starts, but it has no carrier until a VM connects to it. systemd-networkd-wait-online, in its default configuration, waited for all the managed interfaces to be online, and that one never would be with the machines powered off. Hence the 35 seconds.

With no <forward> element, the network has no way out: the machines can see each other and the host, and nothing else. It is what you use to test a firewall configuration or to analyse something suspicious.

The bridge is the mode you need when the VM has to be a reachable server. It requires reconfiguring the host's network with netplan, applying 06-01 — and with the same care, because it can leave you with no access to the machine:

That netplan try is not optional. Configuring a bridge over SSH is exactly the case it exists for: if the bridge goes wrong, you lose the session, and the automatic rollback after 120 seconds gives the machine back to you.

And a specific warning: the bridge moves the physical interface's IP onto the bridge. During the transition there is a network outage of a few seconds, and if the configuration is wrong the outage is permanent. On a remote server with no console, this is done with enormous care.

The devices QEMU presents to the guest can be emulated (imitating a real device, such as an Intel e1000 card) or paravirtualised (virtio: a device that does not exist in the real world, designed to talk directly to the hypervisor).

The difference is large and worth seeing measured:

The clues that virtio is active: the disk is called vda (not sda) and the bus is virtio. If you see sda with a sata bus, you are using emulation and losing more than half of the disk performance.

The three attributes of the <driver> matter as well:

discard=unmap is what stops a qcow2 disk growing indefinitely: without it, deleting data inside the guest frees no space on the host.

The three modes:

And here the outstanding item from 07-03, the AES-NI one, gets resolved:

4.4 GB/s of AES encryption: that is only possible with hardware acceleration. With host-model instead of host-passthrough, or with a generic virtual CPU, the same openssl speed would give between 200 and 600 MB/s, and there is the factor of five to ten that was put forward as a hypothesis in the last exercise of 07-03 for the slowness of the encrypted backup.

The downside of host-passthrough is live migration: a machine that sees a processor's exact extensions cannot be migrated to another host that does not have them, because the guest is using them. For a test environment on a single host, host-passthrough is the right choice.

Two more memory settings, mentioned for completeness:

Ballooning lets the host reclaim memory from a guest that is not using it, through a driver inside the guest that "inflates" it. It is useful for density, and counterproductive for a database, which reserves shared memory and does not expect to lose it. Hugepages (<memoryBacking><hugepages/></memoryBacking>) reduce the guest's TLB misses, with the same logic as the THP from 07-03, and they are common on large database machines.

The cloud-init machine is clean and reproducible, but to reproduce a production problem you sometimes need a copy of the real state. virt-clone makes one:

virt-clone copies the disks and generates new MAC addresses and UUIDs, which is what avoids the conflict caused by copying the file by hand. But it leaves the inside of the guest untouched: the same machine name, the same SSH host keys, the same machine-id. And that causes real problems:

virt-sysprep cleans out what makes a machine unique: the machine-id — which systemd and the journal use as an identifier — SSH host keys, logs, shell history, and network configuration tied to the previous MAC. Without this step, two machines with the same machine-id produce mixed-up journal entries if they are centralised, and two with the same host keys trigger the changed-fingerprint warning from 06-02.

And libguestfs also lets you work with a powered-off machine's disk without starting it, which solves the 07-01 scenario without needing rescue media:

That is what turns the recovery of 07-01 into a two-minute operation when the machine is virtual: instead of booting from an ISO, mounting and doing a chroot, you edit the file directly on the powered-off disk.

With srv-tramontana-test up and running, the four debts from the start of the lesson can be settled:

On live migration, to close: libvirt lets you move a running machine from one host to another without interrupting it, copying the memory in several passes until so little is left to transfer that it can be paused for a few milliseconds and the switch completed.

It requires shared storage (or --copy-storage-all, which is much slower) and compatible CPUs — hence the downside of host-passthrough. It is the basis of maintenance with no outage: the hosts are drained one by one to update them. And it is a preview of 07-07: moving a machine is not the same as having high availability, because migration requires the source host to still be working. Against a sudden failure, it is no use.

Graphical and higher-level tools, for the record: virt-manager is libvirt's graphical interface, convenient for exploring and for reaching the console of a machine from a graphical environment; Vagrant defines development machines in a versionable Vagrantfile and works over libvirt, VirtualBox or VMware, and it is the right tool when a whole team needs identical environments; and Terraform and OpenStack operate at infrastructure scale, also talking to libvirt underneath.

• Not enabling nested virtualisation. kvm-ok says KVM is disabled by your BIOS and everything works by emulation, ten times slower. On VirtualBox, VBoxManage modifyvm <vm> --nested-hw-virt on with the machine powered off.
• Forgetting console=ttyS0 on the kernel command line. virsh console shows a blank screen and it looks as though the machine is not booting. It is paired with --console pty,target_type=serial.
• Using emulated devices instead of virtio. If the guest's disk is called sda rather than vda, you are losing more than half of the disk performance. Check for bus='virtio' in the XML.
• Confusing destroy with deleting. virsh destroy is pulling the cable: it deletes nothing, but it can leave the guest's filesystem inconsistent. Use shutdown and wait.
• Modifying a base image that has disks backed by it. It corrupts every derived disk. The base is read-only in practice; use qemu-img rebase if you need to make one independent.
• Trusting a snapshot as a backup. It lives on the same disk and on the same machine. It meets none of the three requirements of the 3-2-1 rule.
• Accumulating chained snapshots. Every active one degrades writing, and a long chain makes the disk unmanageable. Consolidate or delete.
• Cloning without virt-sysprep. Two machines with the same machine-id and the same SSH host keys cause mixed-up journals and changed-fingerprint warnings.
• Configuring a bridge over SSH without netplan try. A badly defined bridge locks you out of the machine permanently. The automatic rollback after 120 seconds exists for exactly this.
• Leaving out discard=unmap. The qcow2 file grows indefinitely because deleting inside the guest frees no space on the host.
• Enabling ballooning on a database machine. The database reserves shared memory and does not expect it to be taken away. It causes degradation that is hard to diagnose.
• A tip on method. Version the cloud-init user-data in git alongside the scripts. A test machine that can be recreated with one command in two minutes gets used; one that has to be reinstalled by hand gets abandoned the moment it drifts out of configuration.

Design the cloud-init user-data for an srv-tramontana-test machine that reproduces production's security configuration as faithfully as possible, and explain what cannot be reproduced and why. Justify every decision.

Release 3.3.0 will not start. Design the complete procedure for diagnosing it in the test environment, using what you learned in 07-01, 07-02 and this lesson, in such a way that you can repeat the attempt as many times as necessary.

Marta asks whether it would be worth moving Tramontana Bookings to virtual machines managed by KVM on a server of our own, instead of the current VPS. Write the analysis: what is gained, what is lost, and what you recommend.

Solution 1

What CANNOT be reproduced, and why. This is the important part of the exercise, because it determines which tests are valid:

And the three decisions worth justifying because they are counterintuitive:

1. Keeping ptrace_scope = 1 in testing. The temptation is to relax it so as to diagnose comfortably. That would be a mistake: we would then not reproduce the Operation not permitted from 07-02 and the diagnostic tests would be worthless. A test environment that differs in security does not test the security.
2. The same numeric UIDs and GIDs. It is easy to let cloud-init assign whatever it likes, and then the 2770 on the tramontana group does not behave the same way, and the permission tests are invalid. Numeric identifiers are part of the configuration.
3. log_level=debug as the only intentional difference, plus the diagnostic tools. Both are more than there is in production, not less, so they invalidate nothing — and they have to be noted, because the log volume does differ.

The last warning goes in the runbook: this user-data contains the SSH public key and the organisation's identifiers, so it gets versioned in the internal repository, not a public one. And it must never contain a real secret: that is what pass and later provisioning are for.

Solution 2

The key to the design is that the diagnosis must be repeatable: every attempt starts from the same state, so that the differences observed are due to the change and not to leftovers from the previous attempt.

The cold snapshot is deliberate: a --live one would capture the memory and, without --quiesce, could leave PostgreSQL inconsistent. For a point of return, powered off is simpler and more reliable.

Without this verification, a 3.3.0 failure could be confused with a test environment that was already broken.

execsnoop-bpfcc is the informed choice here: if the binary starts and dies in 200 ms, ps will never see it, and this tool captures every execve with its exit code. It is exactly the use case from the table in 07-02.

And here is the decision tree that makes this procedure useful, because each symptom has a different tool:

Step 4e is the one with the best return and the one usually forgotten: comparing the version that works with the one that does not. If the only difference is a new library, you have the answer without tracing anything.

Three design decisions that justify the procedure:

1. The observers are launched before the attempt. A binary that dies in 200 ms cannot be observed after the fact. It is the difference between having the evidence and having to repeat.
2. The decision tree is written before you start. Under pressure, the temptation is to fire strace at everything and drown in output. With the table in front of you, each symptom leads to a specific tool.
3. The snapshot allows unlimited attempts. And that is the reason for the lesson: this diagnosis cannot be done in production. Every failed attempt leaves a half-deployed release, a service down and a rollback. In testing, snapshot-revert and go again.

A final note: when the cause is found, the fix is applied and tested in testing until health_check.sh returns 0, and only then is it deployed to production. deploy.sh having had an automatic rollback since 04-07 avoided disaster at the time; having somewhere to diagnose is what allows it to be fixed.

Solution 3

Analysis: moving Tramontana Bookings to our own virtualisation with KVM To: Marta Vidal · From: Systems Operations · 18 August 2026

Short recommendation: not now. Keep the VPS and use KVM solely for the test environment, which is where the benefit is immediate and the risk is nil. Here is the reasoning in detail.

---

What we would gain

1. Complete control of the stack. We could tune the machine's CPU (host-passthrough), the disk configuration and the I/O scheduler. Remember that in this week's performance tests we could not determine whether the backups' encryption was hardware-accelerated, precisely because we do not control that layer.
2. Strong isolation between services. We could separate the database from the application onto different machines, with a real security boundary between them. Today they share an operating system: a compromise of the application reaches the database directly.
3. Snapshots before every change. A point of return measured in seconds, instead of depending on restoring backups with their 8-hour RTO.
4. Predictable cost above a certain volume. A server of our own with capacity for several machines can work out cheaper than four or five VPSs.
5. No dependence on the provider as regards kernel versions, available features or price changes.

What we would lose, and this is what settles it

1. The hardware becomes our problem. Disks, power supplies, memory, fans. A disk that fails at three in the morning is the provider's to resolve today; with our own server it is ours, and we would need disks in RAID — remembering that RAID is not a backup — and spare parts.
2. Power, cooling and networking stop being guaranteed. A server in an office depends on a single electrical circuit and a single data line. A two-hour power cut is a two-hour outage. Hosting with a redundant UPS and dual feeds costs money.
3. A single point of failure, and a new one: the host. Today we have one server that can fail. With our own virtualisation we would have the virtual machines plus the host, and if the host goes down, they all go down together. This matters: virtualisation does not provide high availability, and believing otherwise is the most common design error in the field. With a single host, the risk of a total outage goes up.
4. The workload increases, it does not decrease. Today we administer one operating system. With our own virtualisation we would administer the host, the hypervisor, the virtual network, the storage and each guest machine. With the current headcount, that is time taken from somewhere else.
5. A real up-front cost. A server with capacity and disk redundancy, plus suitable hosting, plus the spares. It is not negligible, and it has to be compared against the annual VPS cost over several years, not one.
6. Compliance implications. We process guests' personal data. A server of our own makes us responsible for the equipment's physical security, which today belongs to the provider and which appears explicitly in our threat model as out of scope. That would have to be reviewed with the data protection officer, and would probably require full encryption of the root disk and access control to the physical space.

---

What I do propose, and it is already under way

Using KVM for the test environment, on the administration machine itself or on the current VPS. The benefit is immediate and the risk is none:

• This week it has made it possible, for the first time, to diagnose the 3.3.0 release that failed in July and was left unexplained.
• It allows configuration changes and kernel updates to be tested before they are applied in production.
• It allows the runbook's recovery procedure to be rehearsed without touching the real server.
• It is recreated from scratch in two minutes from a configuration file versioned in git.

When to reconsider. The analysis changes if at least two of these conditions are met:

Condition	Why it changes the decision
We need four or more servers	The per-machine cost of our own server drops a great deal
There is budget for two hosts	Only then does virtualisation add availability rather than subtract it
Professional hosting is available	It resolves power, cooling and networking
There is somebody dedicated to infrastructure	The workload stops being taken from somewhere else
The VPS cost clearly exceeds the alternative over three years	That is when the investment genuinely pays for itself

And an intermediate alternative worth considering first. If the real objective is separating the database from the application for security, or having a permanent pre-production environment, taking out a second VPS achieves exactly that for a fraction of the cost and without taking on responsibility for the hardware. It is a reversible step; buying a server is not.

Summary for the decision. Our own virtualisation is the right answer when the problem is scale and there are resources to do it properly. Our problem today is not scale: it is that we had no test environment — now resolved — and that we have a single point of failure, which is a redundancy problem and not a virtualisation one. I am looking into that second matter and will bring you a proposal with numbers.

You now have the test environment the course had spent four modules missing, and you understand the stack that supports it. You know what each piece virtualises: KVM gives access to the CPU's extensions, QEMU emulates the devices, libvirt manages the whole with a stable API, and virsh is the day-to-day tool. You know that destroy is pulling the cable and not deleting, that virsh console is the way to recover a broken boot with no graphical interface, and that virtio on the disk and on the network is the difference between a slow machine and a fast one. You provision unattended with cloud-init from a file versioned in git, you choose qcow2 for its snapshots and backing files knowing what that 2-5% of performance costs, and you are clear that a snapshot is not a backup.

And along the way you have closed three outstanding matters. The virbr0 that cost 35 seconds of boot time in 07-01 now has a complete explanation: an interface that exists but has no carrier until a machine starts. The 07-03 question about AES-NI is resolved: with host-passthrough, the guest sees the CPU's extensions and encrypts at 4.4 GB/s, which confirms the factor of five to ten that was put forward as a hypothesis. And the 3.3.0 release that failed in Module 4 finally has somewhere to be reproduced as many times as necessary, with a snapshot as a point of return and a decision tree written before you start.

Notice now the price you have paid for the strong isolation of virtual machines. Every VM carries its own kernel, takes half a minute to start, occupies gigabytes on disk and reserves hundreds of megabytes of memory simply by existing. For a test environment that is perfectly reasonable. For deploying an application and its database, and being able to recreate them in seconds, it is extremely expensive. And in the diagram in the third section you already saw the alternative: share the host's kernel and isolate only what is necessary.

In lesson 07-05: Linux Containers and Docker that is what gets built, and it starts with the idea you have to be clear about before writing your first docker run: a container is not a small machine, it is an isolated process. You will see the three kernel primitives that make it possible — namespaces, which isolate what a process sees; cgroups v2, which are literally the same technology as the MemoryMax=512M you set in 05-05; and the capabilities from 05-02 — and you will recognise in the mount namespaces the chroot you used to repair GRUB in 07-01. Then comes Docker: images and layers, a Dockerfile with a multi-stage build so that the compiler is not shipped to production, volumes versus bind mounts, networks with name resolution, and a compose.yaml that brings the application and PostgreSQL up together with health checks. With two warnings the lesson takes seriously: belonging to the docker group is in practice equivalent to having root, and that kernel.unprivileged_userns_clone you left commented out in 06-06 with a note explaining why — the moment has come to understand it fully.