We closed Module 5 with an uncomfortable question: for four lessons we have been faking isolation. ProtectSystem=strict fakes a read-only file system, PrivateTmp fakes a private /tmp, capabilities fake a trimmed-down root. All of that works, but all of it is decided by the very same kernel that runs the process we want to contain: if that kernel goes down, everything goes down with it.

Virtualization answers with an idea that was already formalized back in 1974 and that today underpins the whole of cloud computing: giving an entire operating system the illusion that it has a machine all to itself. Not a private directory: its own CPU, its own physical memory, its own disk, its own network card, and its own kernel running in privileged mode. If meteo-api runs inside a virtual machine and someone gets root and breaks that virtual machine's kernel, they still have a whole boundary left ahead of them.

In this lesson you will understand what virtualizing exactly means, why for twenty years x86 was the worst possible architecture for it and the three solutions that were invented, how each of the three resources we studied in Module 2 gets virtualized — CPU, memory and I/O — and what price you pay for all of it, with numbers. We will finish by bringing up a test replica of meteo-01 with KVM and libvirt, and measuring the real overhead. Containers, which are another answer to the same problem with a completely different trade-off, are the next lesson; here we will only say why a virtual machine isolates more.

1. What it means to virtualize a complete machine
2. The historical problem: consolidation, isolation and utilization
3. The Popek and Goldberg criteria
4. Why x86 was not virtualizable, and the three answers
5. Type 1 and type 2 hypervisors
6. CPU virtualization: vCPUs, oversubscription and steal time
7. Memory virtualization: EPT/NPT, ballooning and overcommit
8. I/O virtualization: emulation, virtio and passthrough
9. Virtual networks: bridge, NAT and internal network
10. The virtual disk: formats, snapshots and cloning
11. Live migration
12. Hands-on with KVM and libvirt: a replica of meteo-01
13. Isolation security: hypervisor surface and escapes
14. When to virtualize and when not to

In 01-01 we defined the operating system as an extended machine: it turns awkward hardware into convenient abstractions (processes, files, sockets). A hypervisor does something different and, in a way, more humble: it does not offer new abstractions, but more copies of the same machine.

Hence the vocabulary:

• Host: the physical machine and the software that manages the sharing.
• Hypervisor or VMM (Virtual Machine Monitor): the layer that creates and controls the virtual machines.
• Guest: the operating system running inside a virtual machine, without knowing that it is.

That last sentence is the key to everything: the guest is neither ported nor modified. A Debian installed in a virtual machine is the same Debian you would install on bare metal. It runs its own init process, its own scheduler, its own page tables and its own drivers. It believes /dev/vda is a disk. It believes it has 4 CPUs. It believes a motherboard performed the BIOS boot.

Notice the essential difference with what we will see in 06-02: here there are two independent guest kernels, each with its own memory management, its own scheduler and its own process table. In a container there is only one kernel, the host's, shared by everyone.

Virtualization was not born out of theoretical elegance, but out of money. In the late 1990s and early 2000s the dominant model was one service, one physical server, for a very sensible reason you already know from Module 5: if two applications share a machine, they share a kernel, they share a file system and they share a fate. A failure in application A took down application B.

The result measured in real data centers of the era was devastating: average CPU utilization between 5% and 15%. That is, machines were bought, powered, cooled and administered while sitting idle 90% of the time, because they had to be sized for peak load and isolated for security.

Applied to Meteora, the arithmetic is easy to see:

Virtualization simultaneously solved three things that until then had been in conflict:

• Consolidation. Many workloads on less iron, with direct savings in purchasing, rack space, electricity and cooling.
• Isolation. Without giving up separation: each workload still has its own operating system, and a guest kernel that panics does not drag the others down with it.
• Utilization and flexibility. Resizing a virtual machine means editing an XML file; adding a disk is one command; cloning an entire server takes seconds. With iron, all of that is weeks and a purchase order.

On top of that came a fourth benefit that turned out to be almost as important: the machine became a file. A full backup, a snapshot before a risky update, a test environment identical to production, moving a workload from one server to another without powering it off... none of that is possible unless the "server" is a set of files and a bit of state.

In 1974, Gerald Popek and Robert Goldberg published the paper that still defines what a real hypervisor is. They laid down three properties it must satisfy:

The third criterion is what separates virtualization from emulation. QEMU in pure emulation mode can run an ARM binary on x86 by translating every instruction: it satisfies equivalence and control, but it is glacially slow. A hypervisor must get the guest's code to run on the real silicon, and only step in at the dangerous moments.

To formalize when that is possible, Popek and Goldberg classified an architecture's instructions. Recall from 01-06 the distinction between user mode and kernel mode:

• Privileged instructions: they raise an exception (trap) if executed in user mode. For example, loading the page table or disabling interrupts.
• Control-sensitive instructions: they change the state of the system's resources (enabling interrupts, modifying the descriptor table, switching mode).
• Behavior-sensitive instructions: their result depends on the state of the system, for example on which privilege ring execution is taking place in.

And then they stated their theorem, which has a beautiful simplicity to it:

A hypervisor can be built for an architecture if the set of sensitive instructions is a subset of the privileged instructions.

The mechanism behind it is called trap-and-emulate, and it works like this:

1. The guest operating system runs deprivileged: not in the ring 0 it believes it has, but in a less privileged ring.
2. While it executes harmless instructions (arithmetic, jumps, accesses to its own memory), it runs at native speed. This is where efficiency is satisfied.
3. When it executes a sensitive instruction — which by hypothesis is privileged — the CPU raises an exception that the hypervisor catches.
4. The hypervisor emulates the effect of that instruction on the guest's virtual state and hands control back. This is where equivalence and resource control are satisfied.

If any instruction is sensitive but not privileged, the whole thing falls apart: the guest executes it without anyone noticing and obtains a result that reveals or alters the machine's real state.

In 2000, John Scott Robin and Cynthia Irvine published an analysis of the Intel Pentium instruction set and found 17 sensitive, unprivileged instructions. x86 failed the theorem.

The canonical example is POPF (pop flags): it pops a value off the stack and loads it into the flags register, which includes the IF interrupt-enable bit. In kernel mode, POPF changes IF. In user mode, POPF silently ignores that bit and raises no exception. It is sensitive (it affects the configuration) but not privileged (it does not trap). A deprivileged guest kernel that tries to disable interrupts with POPF believes it has done so, and nothing has happened. From then on the guest misbehaves in ways that are impossible to debug.

Another, even more direct example: SGDT, SIDT, SLDT and STR read the descriptor table registers and work in user mode. A guest can read them and discover that its tables are not where it put them: it knows it is virtualized, and worse, it sees real host data.

The industry gave three answers, chronologically and with opposing philosophies.

VMware Workstation solved the problem without touching the hardware or the guest: instead of directly executing the guest kernel's code, it translates it on the fly into an equivalent, safe block that is kept in a translation cache.

• The guest's user-mode code runs directly, untranslated: it is not sensitive.
• The guest's kernel-mode code is translated block by block. Harmless instructions are copied as they are; the 17 problematic ones are replaced with calls into the hypervisor.
• Translated blocks are cached, so a loop is translated once and executed thousands of times.

It was a feat of engineering and it worked, with a penalty of 5% to 20% depending on the workload; its cost was brutal complexity and a lot of sensitivity to workloads with plenty of kernel code.

Xen took the opposite route: if the problem is that certain instructions do not trap, let us not use them. The guest operating system is modified so that, wherever it would perform a privileged operation, it explicitly calls the hypervisor through a hypercall — exactly the same concept as a system call from 01-06, but one level further down.

Advantages: excellent performance (1% to 5% overhead) and a simpler design. Decisive drawback: the guest has to be modified. With Linux that was viable (open source), with Windows it was not. Paravirtualized Xen required specific kernels.

Paravirtualization disappeared as a CPU virtualization technique, but it survived with enormous success in I/O: virtio, which you will see further down, is exactly this idea applied to disks and network cards, and today it is the de facto standard.

The definitive solution was to fix the architecture. Intel VT-x and AMD-V (SVM) added a new privilege axis perpendicular to the rings:

• VMX root mode: where the hypervisor runs. It has its own four rings 0-3.
• VMX non-root mode: where the guest runs. It also has its own four rings 0-3.

This is the important part and the one usually explained badly: the guest kernel runs in ring 0, just as it expects, but in non-root mode. It is not deprivileged. Nothing has to be translated. And in non-root mode, sensitive instructions do cause an exit to the hypervisor, including the 17 traitors. Popek and Goldberg's theorem is satisfied by decree of the silicon.

Root mode is colloquially called "ring -1", the expression we already hinted at in 01-06: a privilege level below the guest's ring 0.

The state of each virtual machine lives in an in-memory structure called the VMCS (Virtual Machine Control Structure) on Intel, or VMCB on AMD. It contains three things:

• The guest state: registers, page table pointers, control state. Saved on exit and restored on entry.
• The host state: where to return to when the guest exits.
• The control fields: which events must cause an exit. Here you decide, for example, whether CPUID exits to the hypervisor or whether writes to CR3 are handled on their own.

The lifecycle is a loop between two instructions and an event:

The VM exit is the unit of cost of all modern virtualization, and it is worth keeping the number in your head:

On a 3 GHz CPU, 1,200 cycles are 0.4 microseconds. It sounds like nothing, but if a workload causes 100,000 exits per second, it is spending 4% of an entire core just entering and leaving. That is why the whole design of modern virtualization consists of eliminating exits: EPT eliminates them for page faults, virtio reduces them by batching I/O operations, and SR-IOV eliminates them entirely for networking.

The classic classification, also from Goldberg, distinguishes according to what sits below the hypervisor.

In practice this boundary has become blurred, and KVM is the perfect example. KVM (Kernel-based Virtual Machine) is not a separate hypervisor: it is a Linux kernel module (kvm.ko plus kvm-intel.ko or kvm-amd.ko) that turns the Linux kernel itself into a type 1 hypervisor.

The move is extremely elegant, and it fits everything we saw in Module 2: Linux already knows how to schedule tasks onto CPUs, it already knows how to manage virtual memory, it already knows how to talk to disks and network cards. A hypervisor needs exactly that. Why rewrite it? KVM only adds what is missing: managing the VMCS and root mode.

With KVM, a virtual machine is an ordinary Linux process, and each vCPU is a thread of that process. This has consequences you can verify:

What it shows. nlwp 7 indicates seven threads: four vCPUs and three auxiliary ones. The %CPU of 185 means the VM is using the equivalent of 1.85 physical cores. And since it is an ordinary process, all the tools from this course apply to it: nice and chrt for its priority (02-02), cgroups to cap its memory and its CPU (06-02), taskset to pin it to specific cores, and even the OOM killer can kill it, with the dramatic effect of abruptly powering off an entire virtual server.

Under KVM, the usual companion is QEMU, which provides what KVM does not do: emulating the motherboard, the BIOS/UEFI, the clock, the PCI bus and the devices. The division of labor is clean: KVM executes the guest's instructions at native speed; QEMU handles the VM exits that correspond to devices.

Each vCPU of a virtual machine is, under KVM, a thread of the QEMU process. And like any thread, it is scheduled by the host's scheduler, the CFS-EEVDF we studied in 02-02. There are therefore two overlapping levels of scheduling:

1. The guest's scheduler distributes its processes across its 4 vCPUs. It believes they are real CPUs and always available.
2. The host's scheduler distributes the vCPUs of all the virtual machines across the 16 physical cores.

Since the guest knows nothing about the second level, it can make bad decisions: for example, spinning on a spinlock (03-04) waiting for another vCPU that the host has preempted and that will not run again for 10 milliseconds. This problem, lock holder preemption, is why modern kernels use paravirtualized spinlocks that yield to the hypervisor.

To oversubscribe is to assign more vCPUs in total than there are physical cores. With 16 physical cores and 6 virtual machines of 4 vCPUs each, there are 24 vCPUs over 16 cores: a ratio of 1.5:1.

It works for the same reason multiprogramming works: hardly any workload uses its CPU all the time. meteo-api spends most of its time blocked in accept() waiting for requests.

Memory, by contrast, is not oversubscribed with the same cheerfulness, because a process that is not using CPU still occupies its RAM.

How does a guest know it is waiting for CPU? The hypervisor tells it. Linux exposes steal time: the time during which a vCPU was ready to run but the host did not give it a physical core.

What st means. That 15.0 st is the single most important figure for diagnosing a slow virtual machine. It says that 15% of the CPU time the guest thought it had was taken by somebody else. And the practical consequence is brutal for troubleshooting:

• If inside the VM you see high latencies with low %us, low %wa and high st, the problem is not inside your machine: it is on the host, which is oversubscribed. Nothing you optimize in your code is going to fix it.
• A sustained st above 5% is already a sign of contention; above 10% it is a problem.
• In the public cloud, high st on cheap instances is often the product you bought: "burstable" CPU instance types work exactly like that.

Three more pieces worth knowing: pinning ties each vCPU to a specific physical core, eliminates migration between cores, preserves cache and NUMA locality and reduces jitter, which makes it the norm for latency-sensitive workloads; masked CPUID filters the capabilities the CPU advertises so that a VM does not discover instructions (AVX-512, for example) that will not exist on the destination if it is migrated; and the clock must be paravirtualized (kvm-clock), because a guest that gets preempted cannot trust counting cycles.

This is, conceptually, the most beautiful point of the lesson, and it builds directly on the paging from 02-04.

Recall the normal scheme: a process uses virtual addresses, and the MMU translates them into physical addresses by walking a four-level page table, with the TLB caching the result.

Now a new problem appears: the guest has its own page tables and translates from guest-virtual to guest-physical. But the guest's "physical" is not really physical: it is a fiction the hypervisor must translate into host-physical, or machine, addresses. There are two chained translations:

Before the hardware helped, the hypervisor maintained shadow page tables: real tables that translated directly from guest-virtual to real-physical, combining the two translations up front. The MMU used the shadow table and never knew the difference.

The cost was terrible: the hypervisor had to mark the guest's tables read-only in order to find out about every modification. Every time the guest created or modified an entry — something an operating system does constantly — there was a page fault, a VM exit and a synchronization job. On workloads with lots of process creation, the penalty could exceed 40%.

Intel EPT (Extended Page Tables) and AMD NPT/RVI solved the problem by adding a second page table to the MMU, managed by the hypervisor, translating from guest-physical to real-physical. Now the MMU does both translations by itself, with nobody's intervention.

The advantage is enormous: the guest can modify its page tables freely, with no VM exits. The guest's page faults are resolved by the guest. The hypervisor only steps in when the second table misses.

The price is the number of memory accesses when the TLB misses. With four-level paging, a normal walk costs 4 accesses. With two nested levels, each of those 4 accesses in turn requires its own 4-level walk:

That is why the practical advice from 02-04 becomes more important inside a virtual machine: using 2 MB huge pages reduces the levels of both tables and multiplies the TLB's reach. On a virtualized database, enabling hugepages can give between 5% and 15% improvement.

A guest that has used memory does not give it back: its kernel keeps it in its page cache, because from its point of view it is free. The host sees that memory as used even though nobody needs it.

The solution is clever and is called the memory balloon (balloon driver). It is a paravirtualized driver inside the guest that, at the hypervisor's request, asks its own kernel for memory and uses it for nothing at all. As it "inflates", the guest kernel comes under memory pressure and frees caches or swaps, exactly as we studied in 02-04; the pages the balloon obtains are reported to the hypervisor, which reassigns them to other virtual machines. When it deflates, the balloon releases them and the guest gets memory back. It is a way for the hypervisor to ask the guest for cooperation instead of blindly stealing its pages.

KSM (Kernel Samepage Merging) scans memory looking for identical pages across different virtual machines and merges them into a single physical copy marked copy-on-write (the same COW from fork we saw in 02-01). If ten VMs run the same Debian, their in-memory binaries are identical and a lot is saved: on homogeneous farms, savings of 30% to 50% have been measured.

Two serious warnings. It costs CPU: the ksmd daemon scans continuously, and is controlled with /sys/kernel/mm/ksm/pages_to_scan and sleep_millisecs. And it has security implications: merging opens a covert channel measurable by timing — writing to a merged page takes longer, because it has to be copied — which lets one VM deduce which pages another one has. In multi-tenant environments, KSM is turned off.

Just as with CPU, you can promise more RAM than exists: 8 VMs of 8 GB on a 48 GB host. It works as long as the VMs do not use everything promised at the same time.

The risk is qualitatively worse than with CPU. If CPU is short, everything is slow. If RAM is short, the host starts swapping guest pages, and that is catastrophic: the guest does not know its "RAM" is on a disk, so its own memory management decisions become absurd — it may be swapping its processes out at the same time as the host is swapping it out, the phenomenon known as double paging. And if memory runs out completely, the host's OOM killer kills a QEMU process: an entire virtual machine vanishes at once, as if someone had pulled its power cable.

Practical rule: overcommit CPU yes, while watching steal time; overcommit memory only with a wide margin, ballooning active and monitoring in place, and never on critical machines.

Here we pick up 02-07 again: drivers, interrupts and DMA. There are three strategies, from worst to best performance.

QEMU emulates a real, well-known device: an Intel e1000 network card, an IDE controller, a Cirrus graphics card. The guest uses its standard driver, knowing nothing.

Its advantage is absolute compatibility: any operating system, however ancient, boots. Its cost is disastrous: every access to a device register — every outb, every write to mapped memory — causes a VM exit, and sending a network packet can cost 10 or 15 exits.

Here Xen's idea comes back to life. virtio defines an explicitly virtual interface: the guest knows it is virtualized and uses a driver designed to talk to a hypervisor as cheaply as possible.

The central mechanism is the virtqueue: a ring of descriptors in memory shared between guest and hypervisor, with the same philosophy as the DMA rings from 02-07.

1. The guest writes several requests into the ring. With no VM exit at all: it is shared memory.
2. When it wants to notify, it issues a single notification (kick), which does cause an exit.
3. The hypervisor processes the whole batch and signals completion with a virtual interrupt.

That "one exit per batch instead of one per operation" is the entire trick, and it is exactly the same amortization logic as NAPI in 02-07. The usual devices are virtio-net, virtio-blk, virtio-scsi and virtio-balloon. On Linux they have been in the kernel for more than fifteen years, and that is why the disks of a modern VM are called /dev/vda and not /dev/sda.

One further turn of the screw is vhost: moving ring processing out of user space (QEMU) into the host kernel (vhost-net), eliminating additional context switches.

The extreme option: giving the virtual machine the physical device. The guest talks to the real hardware with its native driver, with no intermediate layer.

This is only safe thanks to the IOMMU we studied in 02-07. Without it, the device does DMA to real physical addresses and a VM could program it to write into the host's memory: a total escape. With an IOMMU (VT-d or AMD-Vi), the device has its own address translation and can only touch its VM's memory. As we anticipated back then, this is the piece that makes passthrough viable.

The problem with pure passthrough is that a device is given to one single VM. SR-IOV (Single Root I/O Virtualization) solves that in the hardware itself: a compatible network card presents itself as one physical function (PF) and up to 64 or 128 virtual functions (VFs), each with its own queues and its own MAC address. Each VF is assigned to a different VM, and the sharing is done by the card's silicon.

The migration column is what decides in practice: passthrough ties the VM to that specific hardware, and with it you lose the flexibility that is half the value of virtualizing.

The hypervisor implements a virtual switch in software, to which the VMs' interfaces are connected. There are three basic topologies.

For a replica of meteo-01 that must receive readings from the stations, bridge mode is the right one: the VM appears on the network as just another machine, with its own IP.

And here is a security warning that connects with 05-03: the host's firewall does not see traffic between VMs on the same bridge. Two virtual machines connected to the same bridge talk to each other at the link layer without going through the host's nftables. The consequences:

• Each guest's firewall is still mandatory; a perimeter one is not enough.
• To filter at the bridge you need specific mechanisms (ebtables, nftables in the bridge family, libvirt filters or the provider's security groups).
• Segmenting with separate virtual networks by function is better than trusting rules: the management network must not share a bridge with the data network.

A virtual machine's disk is normally a file on the host, though it can also be an LVM volume (04-03) or a LUN on a storage array.

Thin provisioning means a disk declared as 500 GB initially occupies a few megabytes and grows as it is written to. It saves an enormous amount of space, but it introduces a real operational risk: you can create ten 500 GB disks on a 2 TB datastore, and the day they all grow, the datastore fills up and every VM stops at once. It is exactly the same overcommit as before, applied to disk, and it requires the same vigilance.

A snapshot freezes the disk state at an instant. In QCOW2 it is implemented by creating a new file that only stores modified blocks and chaining it on top of the previous one, which becomes read-only.

This has two consequences that are always underestimated. The first is that every read may require walking the chain: with 5 chained snapshots, reading a block can mean looking in 5 files, and performance degrades cumulatively. The second is that a snapshot is not a backup: it depends on the base file, so if the original disk becomes corrupted the snapshot is useless, and if it is kept on the same storage a datastore failure takes both away. The 3-2-1 backups from 05-03 are still mandatory.

The operational rule is clear: snapshots are for short windows — "I am about to update the kernel, if it fails I roll back" — and they are consolidated or deleted within hours, not months.

A template is a prepared base image (system installed, updated, hardened and "cleaned" of unique identifiers, with virt-sysprep). From it:

• Full clone: the entire disk is copied. Independent, takes up its full size, takes time.
• Linked clone: a new QCOW2 is created with the template as a read-only base. It occupies megabytes and is created in a second; only the differences are written.

Pause a moment on the linked clone, because it is exactly the same idea you will see in 06-02 with container images and overlayfs: a shared read-only base layer and one write layer per instance. The technique is so good that it shows up in both worlds.

Moving a running virtual machine from one host to another, without powering it off and with an interruption of tens of milliseconds, is probably virtualization's most impressive feature, and the one that makes it viable to maintain a data center without downtime windows.

What has to be moved is: the CPU state (registers, a few kilobytes: trivial), the device state (small), the disk (if storage is shared, nothing to copy: this is the reason SANs and storage arrays exist) and the memory, which is the challenge: 16 GB over a 10 Gbit/s network is about 13 seconds at best, and during those 13 seconds the VM keeps modifying pages.

The standard technique is iterative pre-copy:

1. Round 0: all memory pages are copied while the VM keeps running.
2. Rounds 1..n: only the pages dirtied during the previous round are copied (the hypervisor tracks them with "dirty" bits).
3. Each round is smaller than the previous one, because there is less time to dirty pages.
4. Final stop (stop-and-copy): when the pending set falls below a threshold, the VM is paused, the last pages and the CPU state are copied, and it is started on the destination. This is the only real interruption: typically 50-300 ms.

The pathological case is called a hot working set: if the VM dirties pages faster than the network copies them — a database with heavy writes — the rounds do not converge and the migration never finishes. The solutions are to throttle the guest's CPU so it dirties less, or to switch strategy to post-copy: move the VM immediately and fetch the pages from the source on demand, which reduces total time but introduces a serious risk — if the network goes down halfway, the VM ends up split across two machines and is lost.

Requirements to keep in mind: compatible CPUs between source and destination (hence the CPUID masking), shared storage or added disk migration, a common network, and no passthrough, for the reason we already saw.

libvirt is the management layer that unifies dealing with different hypervisors (KVM, Xen, LXC, ESXi) behind a single API, a daemon (libvirtd) and a command-line tool (virsh). It defines each VM as a domain XML.

First, check that the machine can virtualize:

What it does. vmx/svm in /proc/cpuinfo indicates VT-x or AMD-V support; if it does not appear, it is almost always disabled in the BIOS, not absent. virt-host-validate walks through all the requirements — modules, /dev/kvm permissions, cgroup controllers, IOMMU — and is the first thing to run when something will not start.

Now we create the test replica. The idea is to have a meteo-01-test identical in configuration to the production one, where updates and restores can be rehearsed without touching the real service:

Option by option, and why:

• --memory 4096 and --vcpus 2,maxvcpus=4: 4 GB and 2 vCPUs, with headroom to reach 4 while running without recreating the VM.
• --cpu host-passthrough: exposes the host's real CPU as it is, with all its instructions. It gives maximum performance and prevents migrating to a host with a different CPU. For production with migration you would use a generic model.
• Two disks: one for the system (60 GB) and another for data (200 GB) for /var/lib/meteora. The separation is the same decision as in 04-03: filling up the data must not prevent booting.
• bus=virtio: the paravirtualized disks we talked about; they will appear as /dev/vda and /dev/vdb.
• cache=none: critical for integrity. It disables the host's page cache for that disk, so that an fsync() from the guest (04-05) really reaches the hardware. With cache=writeback it is faster, but a power cut on the host can corrupt the guest's file system and render journaling useless.
• discard=unmap: propagates the guest's TRIM to the QCOW2 file, so that deleting inside the VM actually frees space on the host. Without this, a thin disk only ever grows.
• --network bridge=br0,model=virtio: bridged so the replica receives readings from the stations, with a paravirtualized NIC.
• --graphics none + --console pty,target_type=serial: no desktop; a serial console, which is what you want on a server and what lets you see the boot messages.

And the everyday operations:

Two important warnings. virsh destroy does not delete the VM: it powers it off abruptly, the equivalent of pulling the cable, with the corresponding risk of corruption; deleting the definition is virsh undefine. And virsh shutdown requires the guest to have qemu-guest-agent installed or at least to respond to ACPI; without that, nothing happens and some people wrongly conclude that the VM has hung.

For initial provisioning without an interactive installer, the standard is to start from an official cloud image and configure it with cloud-init: users, SSH keys, packages and files are declared in a YAML that is injected on first boot. It is the tool that turns a generic template into an already-configured meteo-01-test, and we will see it with a complete example in The Operating System in the Cloud.

Virtualization's promise is that a total compromise of the guest does not reach the host. How true is that?

The hypervisor's attack surface is everything a malicious guest can touch:

The historical case everybody cites is VENOM (CVE-2015-3456), an overflow in QEMU's virtual floppy controller. A guest with root could escape to the host, and the lesson was humiliating and hugely valuable: the vulnerability was in a device that nobody had used for twenty years and that was there "for compatibility". It is the minimal-surface principle from 05-01, applied to the VM definition: remove the floppy, the USB, the audio and the VGA from your servers.

Host hardening measures are, at bottom, the same ones from Module 5 applied to the QEMU process: it runs as the unprivileged libvirt-qemu user, with sVirt (SELinux or AppArmor) labeling each domain so that a compromised QEMU cannot touch another VM's disks, with seccomp filtering its syscalls and with the IOMMU bounding DMA. Defense in depth does not disappear because you virtualize: it is applied one level further down.

This is the point to take away from the lesson, and the one that sets up the next:

• In a virtual machine, the boundary is the hypervisor. An attacker who gets root in the guest and breaks the entire guest kernel still has ahead of them a narrow, heavily audited interface: a handful of VM exit handlers and a few virtio devices.
• In a container, the boundary is the host's kernel, and its interface is the system call table: between 300 and 400 entry points, many with a history of flaws.

The quantitative comparison is brutal: hundreds of syscalls versus a handful of handlers. That is why, when you have to run untrusted third-party code — the typical case of the public cloud, or of a system that compiles customers' code — the right answer is still a virtual machine, or one of the intermediate technologies we will see in 06-03.

The real overhead, measured on modern KVM with virtio and EPT:

With those numbers in hand:

Virtualize when you need strong isolation between workloads, to run different operating systems, to consolidate underused servers, to be able to live-migrate so as to maintain the iron without downtime, to have snapshots and test environments identical to production, or to run code you do not control.

Do not virtualize when the workload needs all the hardware (a database squeezing an NVMe, a scientific compute node), when latency is critical down to the microsecond (trading, telecommunications without SR-IOV), when hardware access is special (data acquisition cards, GPUs without virtualization), or when the real goal is to package and deploy an application: containers are incomparably lighter for that, and that is precisely the next lesson.

And a piece of architectural advice: it is not a binary choice. The most widespread pattern today is containers inside virtual machines: the VM provides the strong security boundary between tenants or environments, and the container provides density and deployment speed within that boundary.

Believing virtualization is "slow". It was in 2004, with binary translation and shadow page tables. Today a pure CPU workload loses 1-3%. What does still cost is badly configured I/O: using emulated devices instead of virtio can multiply the network's CPU consumption by five.

Ignoring st in top. It is the first indicator to look at on a slow VM and the only one that says "the problem is not in here". Many hours of optimization have been wasted tuning an application when what there really was was an oversubscribed host.

Overcommitting memory the way you overcommit CPU. They are not equivalent. A CPU shortage is slowness; a RAM shortage is the host's OOM killer killing an entire VM. Always leave headroom for the host itself (about 2-4 GB) and monitor.

Using cache=writeback in production. It is tempting because benchmarks improve a lot, but it breaks the durability guarantee of fsync() we studied in 04-05: the guest believes it has written and the data is in the host's RAM. A power cut corrupts the guest's file system. Use cache=none (or directsync) for data that matters.

Accumulating snapshots. Every link in the chain adds indirection to every read, and a chain of ten snapshots can halve performance. And they are not backups: they depend on the base file.

Confusing virsh destroy with deleting. destroy is "pull the cable"; undefine is deleting the definition. Learning this the hard way on a production domain is a rite of passage worth avoiding.

Leaving unnecessary emulated devices in place. The floppy, the USB, the audio and the graphics card of a virtual server only add attack surface (remember VENOM). Review the domain XML and remove everything you do not use.

Forgetting the firewall inside the VM. Traffic between VMs on the same bridge does not go through the host's nftables. Every guest needs its own policy, with policy drop as in 05-03.

Tip: always measure inside and outside. When a VM is slow, compare its metrics with the host's (vmstat, the host's iostat, virsh domstats). Half of all performance problems in virtualization are diagnosed by looking at the wrong level.

Tip: enable huge pages for memory-heavy workloads. With EPT, a TLB miss can cost up to 24 memory accesses. 2 MB pages drastically reduce that pressure, and on virtualized databases the improvement is measurable.

A fictitious architecture, NOVA-32, has these instructions:

(a) Classify each instruction as privileged, sensitive (and of which kind) or harmless. (b) Does NOVA-32 satisfy the theorem? Justify your answer. (c) For each problematic instruction, describe a concrete scenario in which it breaks equivalence or resource control. (d) Propose the three historical solutions applied to this case and say which you would choose if the guest is a proprietary system whose source code you do not have.

A host has 16 physical cores, 64 GB of RAM and an NVMe. It has to accommodate: meteo-01 (production: 8 vCPUs, 24 GB), meteo-01-test (4 vCPUs, 8 GB), a continuous integration server (4 vCPUs, 8 GB, bursty), a metrics server (2 vCPUs, 4 GB) and two lab VMs (2 vCPUs, 4 GB each).

(a) Compute the CPU and memory oversubscription ratios, and say whether they look acceptable to you, justifying it. (b) Inside meteo-01 you observe %Cpu(s): 9.1 us, 2.0 sy, 71.0 id, 0.6 wa, 17.3 st: interpret the diagnosis and say which three concrete actions you would take, in order. (c) What disk and network configuration would you give meteo-01 versus the lab VMs, and why? (d) Would you enable KSM? Reason your answer considering the expected savings and the risk.

Meteora wants the meteo-01 replica to be live-migratable between two hosts so the hardware can be upgraded without stopping the service. The ingestor receives about 8,000 readings per second (24 bytes each) and meteo-api serves HTTPS traffic.

(a) Choose the network I/O strategy and justify it against the other two, taking the migration requirement into account. (b) Estimate the duration of the pre-copy phase if the VM has 24 GB of RAM and the migration network is 10 Gbit/s, and explain what determines whether the migration converges. (c) List the requirements the two hosts must meet for the migration to be possible. (d) Describe what would happen if an SR-IOV VF had also been assigned to the VM, and how you would resolve it.

Solution 1

(a) Classification:

(b) It does not satisfy the theorem. The set of sensitive instructions {LDTAB, RDTAB, SETINT, RDMODE, IOWR} is not contained in the set of privileged ones {LDTAB, SETINT, IOWR}: RDTAB and RDMODE fall outside. With pure trap-and-emulate, those two would execute without the hypervisor ever finding out.

(c) Concrete scenarios:

• RDTAB: the guest kernel loads its page table with LDTAB (which traps and the hypervisor emulates, actually pointing at the shadow table). It then reads it back with RDTAB to check it and gets the address of the host's real table, not the one it wrote. This breaks equivalence (behavior differs from real hardware) and additionally leaks host information, weakening resource control.
• RDMODE: the guest kernel believes it is in ring 0 and checks its level with RDMODE; it gets, for example, "ring 1". A kernel that verifies its own privilege before doing something critical will take an error branch or panic. This breaks equivalence and also lets the guest detect that it is virtualized, which is relevant in malware analysis.

(d) The three solutions applied:

• Dynamic binary translation: the guest's kernel code is scanned and RDTAB and RDMODE are replaced with jumps into the hypervisor, which returns the virtual values the guest expects. It works without touching the guest or the hardware, at the cost of complexity and performance.
• Paravirtualization: the guest kernel is modified so that it does not use RDTAB or RDMODE, but equivalent hypercalls instead. Optimal performance, but it requires the guest's source code.
• Hardware assistance: a non-root mode is added in which RDTAB and RDMODE cause an exit to the hypervisor. It is the clean solution, but it requires revising the architecture.

Choice with a proprietary guest and no source code: dynamic binary translation, which is the only one of the first two that does not require modifying it. If the silicon can be redesigned, hardware assistance is superior in every way (simpler, faster and safer), and that is exactly the path x86 took between 1999 and 2006.

Solution 2

(a) Ratios.

• CPU: 8 + 4 + 4 + 2 + 2 + 2 = 22 vCPUs over 16 cores → 1.375:1. Perfectly reasonable: it is within the conservative range, and several of those workloads (CI, lab) are intermittent.
• Memory: 24 + 8 + 8 + 4 + 4 + 4 = 52 GB committed out of 64 GB. There is no overcommit, but the margin is 12 GB, from which you have to subtract the host itself (2-4 GB) and QEMU's own structures (about 200-500 MB per VM, that is 1.2-3 GB). The real margin ends up around 5-8 GB: tight but viable; it does not allow adding more VMs without overcommitting.

(b) Interpreting the top output. st 17.3 with us 9.1 and id 71.0 is a textbook diagnosis: the guest is not saturated, but it is not getting the CPU it thinks it has. The wa 0.6 rules out disk. Therefore the problem is on the host, not in meteo-01: there is CPU contention from other VMs. Actions, in order:

1. Look at the host, not the guest: vmstat 1, virsh domstats --cpu-total for all the VMs, to identify who is consuming. Most likely it is the CI server, which works in bursts and saturates.
2. Contain the culprit by capping its CPU: virsh schedinfo ci --set cpu_quota=... or, better, CPUQuota via cgroups (06-02). The goal is that the elastic workload must not harm the critical one.
3. Prioritize and pin production: raise meteo-01's CPU weight (cpu_shares) and consider pinning its 8 vCPUs to specific cores, reserving the rest for the others, to eliminate contention structurally instead of reactively.

(c) Differentiated configuration.

The justification is integrity versus speed: in production, cache=none guarantees that fsync() reaches the hardware and that ext4's journaling keeps its promise; in the lab, losing data to a power cut is irrelevant and the linked clone saves space and time.

(d) KSM: no, or with caveats. The expected savings are high because the six VMs share a distribution (Debian stable) and therefore many identical binary pages: you could expect between a 20% and a 40% reduction in consumption, which here would be valuable given the tight margin. But: the host is not overcommitted, so the benefit is not needed right now; ksmd consumes CPU precisely on a machine that is already showing CPU contention; and KSM opens a covert channel between VMs. Decision: do not enable it by default. If overcommitting memory became necessary in the future, it would be enabled with a low pages_to_scan and only between VMs at the same trust level, never if there were third-party workloads.

Solution 3

(a) Network I/O strategy: virtio-net with vhost-net.

• Versus the emulated device (e1000): with 8,000 readings/s plus the HTTPS traffic, emulation would cause on the order of tens of thousands of VM exits per second, at a cost of ~1,200 cycles each; a noticeable fraction of a core would be wasted on exits alone. Ruled out on performance.
• Versus SR-IOV / passthrough: it would give the best performance and the lowest latency, but it prevents live migration, which is the explicit requirement of the exercise. Ruled out on functional requirements.
• virtio + vhost-net leaves the overhead at 5-15% with migration fully supported. And the volume is not demanding: 8,000 × 24 B = 192 KB/s of useful data, ridiculous for any modern link; what matters here is the packet and interrupt rate, which is exactly what virtio amortizes by batching in the virtqueue.

(b) Pre-copy time and convergence. 24 GB = 192 Gbit. At a theoretical 10 Gbit/s, with a realistic 80% utilization (8 Gbit/s), round 0 takes about 24 seconds. The following rounds copy only the pages dirtied during the previous one.

Convergence depends on comparing two speeds: the dirtying rate (pages modified per second × 4 KB) against the migration bandwidth. If meteo-01 dirties 200 MB/s and the network transfers 1 GB/s, each round is five times smaller than the previous one and within 4-5 rounds it drops below the threshold: it converges and the final stop is tens of milliseconds. If meteo-01 were continuously rewriting a large cache in /dev/shm/meteora-cache faster than the network, it would never converge. In that case: increase the migration bandwidth, throttle the guest's CPU (auto-converge), enable compression, or switch to post-copy accepting its risk.

(c) Requirements for the two hosts:

1. Compatible CPU: same vendor and model presented to the guest. With --cpu host-passthrough migration only works between identical CPUs; to really migrate you have to use a common CPU model that masks CPUID down to the lowest common denominator.
2. Shared storage (SAN, NFS, Ceph) reachable by both at the same path; otherwise you have to add disk migration, which multiplies the time.
3. A common network between the two hosts, ideally a dedicated migration network so as not to compete with service traffic, and the br0 bridge defined with the same name on both.
4. Compatible versions of QEMU and libvirt (you migrate forward, not backward), and authenticated communication between the libvirtd daemons.
5. No directly assigned devices and no host-local resources (ISOs mounted from local paths, for example).

(d) With an SR-IOV VF assigned, live migration is not directly possible, because the physical device's state lives inside the card and is not transferable, and because the guest has a driver specific to that hardware loaded. Solutions, from simplest to most complex:

• Do not use SR-IOV and stay with virtio: it is the right decision here, because Meteora's volume does not require it.
• Bonding the VF with a virtio-net inside the guest: before migrating, the VF is detached, traffic continues over virtio during the migration, and on the destination a VF is attached again. It works, but it adds configuration complexity.
• Accept a migration with downtime (power off, move, power on), which fails the requirement.

Virtualizing is giving an entire operating system the illusion of a machine of its own, and the difference with what an operating system does is that the hypervisor does not offer new abstractions, but more copies of the same machine. It was born out of money — utilizations of 5-15% in data centers where a server had to be dedicated to each service in order to isolate them — and it solved consolidation, isolation and flexibility all at once, with a side effect that turned out to be decisive: the machine became a file, and with that came snapshots, templates, cloning and migration.

The theoretical framework is the Popek and Goldberg criteria — equivalence, resource control and efficiency, which is what separates virtualization from emulation — and their theorem: the sensitive instructions must be a subset of the privileged ones in order to apply trap-and-emulate. x86 failed with 17 sensitive, unprivileged instructions, of which POPF is the canonical example: in user mode it ignores the interrupt bit without raising an exception. The three answers were VMware's dynamic binary translation (touching neither hardware nor guest, at the cost of complexity), Xen's paravirtualization (hypercalls, excellent performance, but the guest has to be modified; today it survives in I/O) and the hardware assistance of VT-x and AMD-V, which added root and non-root mode — "ring -1" — with the VMCS holding the state and the VM exit as the unit of cost, some 1,000-1,500 cycles that all later design has been devoted to avoiding.

The classification into type 1 and type 2 is still useful, but KVM blurs it: by being a Linux kernel module, it turns Linux into a hypervisor by reusing its scheduler, its memory manager and its drivers, so that a VM is a process and each vCPU is a thread — with everything that implies: nice, cgroups, taskset and even the OOM killer apply to it. On the three resources: for CPU, two overlapping levels of scheduling, reasonable oversubscription from 2:1 to 4:1 and top's st as the indicator that says "the problem is not in here"; for memory, the double translation solved historically with shadow page tables and today with EPT/NPT, which eliminates VM exits in exchange for up to 24 accesses per TLB miss — hence the importance of huge pages — plus ballooning, KSM with its covert channel and an overcommit far more dangerous than the CPU one; and for I/O, the ladder of emulation, virtio (the default, because it amortizes exits by batching in the virtqueue) and passthrough/SR-IOV with an IOMMU, which gives maximum performance in exchange for losing migration.

Around all that, the practical pieces: networks in bridge, NAT or isolated mode — with the warning that the host's nftables does not see traffic between VMs on the same bridge — disks with thin provisioning and its risk of filling up simultaneously, snapshots that degrade performance and are not backups, linked cloning that anticipates exactly the layering model of containers, and live migration by iterative pre-copy, whose challenge is memory and whose enemy is a working set that gets dirtied faster than the network copies it. In KVM and libvirt, all of that is handled with virsh and virt-install, where the decisions that really matter are bus=virtio, cache=none for integrity and discard=unmap for space.

And the security conclusion, which is the one that connects to what comes next: the hypervisor's surface is narrow but real — VENOM was in the floppy controller nobody used — it is hardened with the same weapons from Module 5 applied to QEMU (unprivileged user, sVirt, seccomp, IOMMU), and it isolates more than a container for a quantifiable reason: the interface a hostile guest can attack is a few VM exit handlers, versus the 300-400 system calls a shared kernel exposes.

That said: all of this has a price that is not the CPU percentage, but the weight. Every virtual machine carries an entire kernel, an init, a complete file system and 200-500 MB of RAM just to exist; it boots in tens of seconds; and its image takes up gigabytes. If what you want is not to run another operating system, but simply to package your application with its dependencies and limit what it sees and what it consumes, you are paying for a kernel you do not need.

And what if, instead of duplicating the kernel, we took advantage of the fact that the kernel you already have knows how to lie to a process about which files exist, which processes there are, which network it sees and how much memory it can use? That does not require a hypervisor: it requires namespaces and control groups, two mechanisms that are already inside Linux and that we have been mentioning since Module 3. That is the next lesson: Containers: Namespaces and cgroups.