Up to now srv-tramontana has always booted. You have installed, configured, hardened and defended a system starting from the premise that, when you switch it on, a prompt appears. This lesson breaks that premise, and it does so in both directions: first you will understand exactly what happens between pressing the button and seeing the prompt, and then you will learn to intervene when that chain breaks.

It is the skill that separates the people who reinstall from the people who fix. An fstab with the wrong UUID, an initramfs that does not include the LUKS module, a GRUB overwritten by another operating system, a lost root password: they are all five-minute problems if you know where to intervene, and an eight-hour reinstall if you do not. And since Module 5 taught you to modify fstab and Module 6 to encrypt a volume with LUKS, you already have on the machine exactly the ingredients that cause these failures.

A note about the lab. Several of this lesson's procedures require the VM console, not an SSH session: when the system does not boot, there is no network. Be clear about how to open the VirtualBox console before you start, and take a fresh snapshot (06-before-boot) — you are going to break the boot process on purpose.

1. The complete boot chain
2. The firmware: UEFI and BIOS
3. GRUB 2: the boot loader
4. The initramfs: the minimal intermediate system
5. The kernel and handing control to PID 1
6. systemd and targets
7. Kernel command-line parameters
8. Recovery: intervening in the boot process
9. Recovering a system that will not boot
10. Reinstalling GRUB from rescue media

Booting a system is a succession of programs, each with more capability than the last, in which every link loads the next one and hands over control. That is precisely why it is called bootstrapping.

The five links and their responsibilities, worth keeping in your head because diagnosis consists of identifying which one broke:

That table is the most useful diagnostic tool in the lesson: when somebody tells you "the server will not boot", the first question is how far did it get.

The firmware lives on the motherboard, runs before anything that is on the disk, and its job is to leave the hardware in a usable state and locate something to boot. There are two generations, and srv-tramontana uses the modern one:

The key difference is the third row. Under BIOS, the loader had to fit in 446 bytes, which forced a two-stage jump and putting code in the space between the MBR and the first partition. Under UEFI, the firmware can read FAT32, so the loader is an ordinary file on an ordinary partition.

That partition is the ESP (EFI System Partition):

Three files worth telling apart, because the order in which they are called matters when something fails:

• shimx64.efi is the first link when Secure Boot is active: it is signed by Microsoft (whose key comes preloaded on the boards), and its only function is to verify and load the next one. It is the bridge between the manufacturer's chain of trust and the distribution's.
• grubx64.efi is GRUB proper, signed by Canonical.
• mmx64.efi (MokManager) manages your own keys, and it is what makes it possible to sign a kernel module of your own (relevant with DKMS, which you will see in 07-03).

The boot entries are not on the disk but in the board's NVRAM, and they are managed with efibootmgr:

BootOrder is the order in which the firmware tries the entries. Being able to reorder it or create a new entry from the running system is what saves the situation when another operating system, or a firmware update, has pushed your entry to the end or deleted it:

A genuine warning: efibootmgr writes to the board's NVRAM, and on some machines — consumer laptops above all — incorrect use has been known to leave the board unusable. On a VM it is completely harmless, and that is where you should practise.

GRUB (GRand Unified Bootloader) solves a problem the firmware cannot: knowing how to read Linux filesystems, understanding LVM and RAID, presenting a menu, and loading a kernel with the right parameters.

The warning is literal and should be taken that way: grub.cfg is generated, and any manual change disappears at the next kernel update, which runs update-grub as part of its maintainer script. The two real sources are:

• /etc/default/grub: the settings in key-value format.
• /etc/grub.d/: the scripts that generate each section of the menu.

The directives you really do touch on a server:

And the operating rule, following the course's convention:

Notice that it finds two kernels. That is not accidental and it is an important safety net: if a kernel update breaks something — a driver that disappears, an incompatible third-party module — the GRUB menu offers you the previous one. The configuration that guarantees it:

Ubuntu keeps the necessary kernels by default and apt autoremove cleans out the old ones. The mistake to avoid is deleting them by hand to free space in /boot when it fills up: if you are left with only one and that one fails, there is nowhere to go back to. The correct solution when /boot fills up is sudo apt autoremove --purge, which respects the running kernel and the previous one.

The scripts run in numerical order and each contributes its part to grub.cfg. 10_linux generates the entries for the installed kernels, 30_os-prober detects other operating systems, and 40_custom is where you put your own entries — because it is the only one that is not regenerated.

To inspect the current menu without rebooting:

This is the least understood link and the one that causes the most problems. The problem it solves is a vicious circle:

To mount the root filesystem, the kernel needs the disk controller's driver, the filesystem module, and — on srv-tramontana — the LVM and LUKS modules, plus the lvm and cryptsetup utilities. But all of that is inside the root filesystem it cannot mount yet.

The initramfs breaks the circle: it is a compressed archive containing a minimal filesystem that GRUB loads into memory alongside the kernel. The kernel mounts it as a temporary root, runs its boot script, which loads the modules, unlocks LUKS, activates the LVM volumes and mounts the real root; and then it does switch_root to move over to it and run the real systemd.

cryptsetup and lvm being there is not automatic: the initramfs-tools scripts include them because they read /etc/crypttab and /etc/fstab at the moment the initramfs is generated. And from that comes the most important operating rule in this section:

After touching /etc/crypttab, the LVM configuration or the boot disk layout, you must regenerate the initramfs. If you do not, the system will boot with an initramfs that knows nothing about the change and will end up at the (initramfs) prompt.

The -k all is the one that saves you: if you only regenerate the current one and later need to boot with the previous one, you meet the same failure from the other kernel.

The configuration lives in /etc/initramfs-tools/:

MODULES=most includes a broad selection of drivers; MODULES=dep includes only those the current machine needs, and produces a much smaller initramfs — at the price of it no longer booting if you move the disk to different hardware. On a stable virtual server it can make sense; as a default value, most is the prudent choice.

When GRUB loads vmlinuz, it decompresses itself and takes control of the hardware for real: it detects CPU and memory, initialises the scheduler, mounts /proc and /sys, loads the built-in drivers and runs the initramfs.

The parameters it was booted with are recorded and can be inspected:

That file is the first thing to look at when the boot behaves unexpectedly: it says what it actually booted with, not what you think it booted with. A single, a nomodeset or an init= forgotten in GRUB_CMDLINE_LINUX shows up here.

And dmesg is the kernel's diary during the boot:

Being able to consult the previous boot is exactly why you made the journal persistent in 05-06. Without /var/log/journal, -b -1 does not exist and the diagnosis of a failed boot is lost when you reboot.

At the end of its initialisation, the kernel runs the init process — today /sbin/init, which is a link to systemd — and hands it control as PID 1. From that moment on the kernel only services system calls; the boot is directed from user space.

You have known systemd since 05-05: units, dependencies, timers. What this lesson adds is its role as the director of the boot.

systemd activates default.target and, working backwards, everything it needs:

The targets you need to know, because they are the ones used in recovery:

And the two boot analysis tools:

The difference between the two is conceptual and it decides where to optimise: blame sorts by how long each unit took, and critical-chain shows the dependency chain that determines the total time. snapd.service takes 2 seconds, but it is not in the critical chain: it starts in parallel and delays nothing. systemd-networkd-wait-online, on the other hand, is, and there it really is worth investigating.

The parameters GRUB passes to the kernel are the lever for intervening in the boot. The ones you need to know:

The first practical recovery tip is the simplest: remove quiet splash. The pretty boot screen hides precisely the messages that would tell you where it broke.

In the GRUB menu, with the entry selected, the e key opens the editor. It is an ephemeral editor: the changes affect this boot only and touch nothing on the disk. That property is what makes it safe to experiment with.

The procedure:

1. Power on the VM and, if the menu does not appear, hold down Shift (BIOS) or press Esc repeatedly (UEFI).
2. Select the Ubuntu entry and press e.
3. Find the line that starts with linux /vmlinuz-....
4. Modify whatever you need at the end of that line.
5. Ctrl+X or F10 to boot with those parameters.

The original line and the three most useful variants:

rescue is the one used 90% of the time: you have the filesystems mounted and the tools available. emergency is for when the problem is in the mounting itself — a broken fstab is the typical case — and there the first thing to do is make the root writable:

init=/bin/bash is the most powerful resource: it replaces PID 1 with a shell, so systemd never runs at all. It is useful when the problem is in systemd or in authentication itself.

And it has two peculiarities that are disconcerting the first time:

PATH is not set, because no startup script has run. You fix it by hand:

And the root is read-only, because the ro parameter is still in force and nobody has remounted it:

When you finish, and this is important: do not reboot with reboot, because there is no systemd to service the request and data may be left unsynchronised. The correct sequence:

A classic scenario, and an uncomfortable demonstration of why physical access — or access to the hypervisor's console — amounts to control of the system:

The security lesson to draw from this: anybody with access to the machine's console can do it in two minutes. The countermeasures — a GRUB password with grub-mkpasswd-pbkdf2, full disk encryption with LUKS including /boot, Secure Boot, and physical access control — are precisely the ones that were left out of scope in the threat model of 06-06. Now you understand why that decision had to be explicit.

You are going to cause it on purpose, because it is the most common failure after a change of disks and because in 06-05 you modified fstab for the LUKS volume.

On booting, instead of the login prompt you get:

This is exactly what mount -a would have prevented. The diagnosis and the repair:

And the two lessons, which go in the runbook:

1. mount -a before rebooting, always. It is a two-second check that avoids this situation.
2. nofail on every mount that is not essential for booting. With nofail, a missing device produces a warning in the journal and the system boots. That is exactly why the backup volume's line carries it:

A different and deeper failure: the initramfs has not managed to mount the root.

The causes, with the check for each one:

If step 4 works, you know the disk and the filesystem are fine and that what is missing is in the initramfs or in crypttab. You boot with the previous kernel from the GRUB menu — whose initramfs is the one from before the change — and regenerate:

It is the reason keeping two kernels is a recovery measure, not a waste of space.

The most serious scenario: GRUB does not appear at all. The firmware finds nothing to boot, or boots straight into another system. It happens after installing another operating system, after cloning a disk, or after a firmware update that wiped the NVRAM entry.

You solve it by booting from external media (the Ubuntu ISO in Try Ubuntu mode), mounting the installed system and entering it with chroot — which is the conceptually interesting operation in this lesson, because it is the same primitive on which the containers of 07-05 are built.

That fourth mount is what makes half of all GRUB repair attempts fail under UEFI: without efivars accessible for writing, grub-install cannot create the boot entry and aborts with an error that does not explain the cause.

Unmounting in reverse order is not a formality: unmounting /mnt before /mnt/dev fails with target is busy, and forcing it can leave the filesystem marked as dirty.

For a system with BIOS/MBR instead of UEFI, step 5 changes and the efivars mounts do not apply:

• Editing /boot/grub/grub.cfg by hand. The file says so on its third line. The changes disappear at the next kernel update. You edit /etc/default/grub and run update-grub.
• Forgetting update-grub or update-initramfs after a change. Editing /etc/default/grub without update-grub does nothing. Touching crypttab or LVM without update-initramfs -u -k all produces the (initramfs) prompt on the next boot.
• Regenerating the initramfs for the current kernel only. If you later need to boot with the previous one, you meet the same failure. Always -k all.
• Deleting old kernels by hand to free up /boot. Being left with only one removes your safety net. Use apt autoremove --purge, which respects the running kernel and the previous one.
• Mounts without nofail in fstab. A missing device stops the machine booting. Only the root and /boot should be able to block the boot.
• Not running mount -a before rebooting. It is the two-second check that separates a warning from a session in emergency mode on the hypervisor's console.
• GRUB_TIMEOUT_STYLE=hidden on a server. A hidden menu cannot be used to intervene. And GRUB_TIMEOUT=0 is worse: it removes the possibility of recovery through the menu.
• Leaving quiet splash in place when diagnosing. Remove them: the messages they hide are exactly the ones that say where it fails.
• Forgetting --bind /sys/firmware/efi/efivars in the chroot. grub-install fails under UEFI with an unhelpful error. It is the most common cause of failed repairs.
• Rebooting with reboot from init=/bin/bash. There is no systemd to service the request. sync, remount read-only, and exec /sbin/init or echo b > /proc/sysrq-trigger.
• A tip on method. Practise the three procedures — a broken fstab, the root password, reinstalling GRUB — in the lab and with a snapshot taken first, today, calmly. The first time you need them will be at three in the morning with a service down, and that is no moment to be learning.

Deliberately cause the (initramfs) prompt in your lab and recover from it. Start from the fact that /srv/tramontana/backups is encrypted with LUKS and its key is declared in /etc/crypttab. Describe the change that causes the failure, why the system does not boot, how you diagnose it from the BusyBox prompt, and the two ways of recovering from it.

srv-tramontana now takes 47 seconds to boot, where before it took 12. Describe the diagnostic procedure using systemd's tools, explaining what information each one contributes and how you tell a slow unit that does not matter from one that does.

Marta asks you for a written boot recovery procedure for the runbook, one that somebody other than you could follow. Write it as a decision tree: what to observe, what to ask yourself and what to do on each branch, covering the four scenarios you have seen in the lesson.

Solution 1

Causing the failure. The realistic scenario is a change in crypttab without regenerating the initramfs. Since the encrypted volume is the backup one and carries nofail, to make the failure a boot failure you have to touch something that does block. The clean way to reproduce it is to remove the LUKS and LVM modules from the initramfs:

Why it does not boot. The initramfs is the only environment that exists before the root is mounted, and its boot script needs cryptsetup to unlock the LUKS volume and lvm to activate the logical volumes. With neither included — because crypttab was empty when it was generated — the script does not find the root device, waits for the maximum time and drops to the BusyBox shell. It is the vicious circle from the initramfs section in its purest form: the tools for mounting the root are inside the root that cannot be mounted.

Diagnosis from BusyBox. Ruling things out from the outside in:

The disk and its three partitions are there. The driver is ruled out.

No output: there is the cause. Neither of the two is in the initramfs.

The system is intact. The problem lies exclusively in the initramfs, and that is the conclusion that guides the repair.

The two ways of recovering from it.

Way A — boot with the previous kernel (the quick one). The initramfs of kernel 6.8.0-39 was generated before the change, so it is still complete... unless you used -k all, which is exactly what we did. If the update-initramfs had covered only the current one, this would be the way out in thirty seconds: GRUB menu → Advanced options → kernel 6.8.0-39 → boot → regenerate. The lesson is twofold: -k all is the correct thing to do so as not to leave a broken kernel behind, but it means a configuration mistake affects every kernel at once. That is why the check (lsinitramfs | grep cryptsetup) goes before the reboot, not after.

Way B — chroot from rescue media (the one that always works).

And the two conclusions that go into the runbook: the .bak-$(date +%F) copies from the course's convention are what made the repair trivial — without them you would have to rebuild the configuration from memory — and the verification with lsinitramfs goes before the reboot, not after. The pattern is identical to mount -a for fstab: checking on the running system what would otherwise only show up at boot time.

Solution 2

The procedure starts by separating the kernel's boot from user space's boot, because they are different problems:

The kernel takes as long as it always did (3.4 s); the extra 35 seconds are in user space. That rules out hardware, drivers and the initramfs, and focuses the search on the systemd units.

blame gives you the suspect, but it is not enough, and here is the substance of the exercise: a slow unit only matters if it is in the critical chain. You have to check:

Confirmed. systemd-networkd-wait-online is in the chain, takes 35 s, and drags along everything that depends on the network: PostgreSQL waits for network-online.target, and tramontana.service waits for PostgreSQL. The 35 s propagate in full to the total.

The contrast the question asks for is provided by snapd.service: it takes 2 seconds and does not appear in the critical chain, because it starts in parallel and nothing waits for it. Optimising it would not save a tenth of a second off the total. That is the difference between the two tools:

The root cause, with the journal:

There it is: virbr0, libvirt's bridge, is configured but has no carrier, because there is no virtual machine running connected to it. By default systemd-networkd-wait-online waits for all managed interfaces to be online, and that one never will be. The 35 seconds are its maximum wait running out.

The fix is to tell it which interface actually matters:

The empty ExecStart= before the new one is compulsory: without it, systemd adds a second command instead of replacing the first, and the unit fails. It is the same list-reset mechanism you saw with SystemCallFilter in 06-06.

And the verification, with the discipline of measuring before and after:

From 47 s to 12.5 s, with the service verified. And a reflection on method: the symptom ("the boot is taking longer") appeared after libvirt was installed, and nobody connected the two things. Recording the boot time in the runbook as part of the 05-07 baseline is what turns "it feels slower to me" into a dated piece of data.

Solution 3

RUNBOOK — Boot recovery for srv-tramontana Version 1.0 · 18 August 2026 · Author: Systems Operations This document is kept OFF the server. A copy on the administration laptop and in the team's password manager.

Prerequisites. Access to the machine's console in VirtualBox (SSH will not do: if it does not boot, there is no network). The root password, available at pass tramontana/production/root. An Ubuntu Server 24.04 ISO available for case 4.

---

STEP 0 — The only question that matters: how far did it get?

Power on the machine and watch. Do not touch anything yet. Note the time and what you see on screen.

What you see	It broke at	Go to case
A black screen, no logo, no menu	Firmware or GRUB	Case 4
The GRUB menu, but it goes no further	GRUB or the kernel	Case 4
The (initramfs) prompt	initramfs	Case 3
Boot messages and then emergency mode	Filesystem mounting	Case 1
It boots but you cannot log in	Authentication	Case 2
It boots, but very slowly	None; it is a slow unit	Case 5

---

CASE 1 — "You are in emergency mode" (the most frequent)

It is nearly always /etc/fstab, after a change of disks.

1. Press Enter and type the root password.
2. Identify what failed: systemctl --failed
3. Read the cause: journalctl -xb | tail -30
4. Make the root writable: mount -o remount,rw /
5. Correct /etc/fstab with nano. If in doubt, comment out the suspect line by putting # in front of it: it is reversible and it gives you the service back.
6. Verify without rebooting: systemctl daemon-reload && mount -a
   • No errors → step 7.
   • Errors → back to step 5. Do not reboot while mount -a is failing.
7. Continue the boot: systemctl default
8. Check the service: /home/operator/scripts/health_check.sh (it must return 0).

Note: if the problem line is the one for /srv/tramontana/backups, it should carry nofail and not block the boot. If it did block it, add nofail as part of the fix.

---

CASE 2 — It boots but you cannot log in

1. Reboot. In the GRUB menu, with the Ubuntu entry selected, press e. (If the menu does not appear: hold Shift or press Esc repeatedly while powering on.)
2. On the line starting with linux /vmlinuz-, change ro to rw and add at the end: init=/bin/bash
3. Ctrl+X to boot.
4. At the bash-5.2# prompt:

   export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
   mount -o remount,rw /
   passwd root                  # or: passwd operator
   

5. If the cause was a change in PAM (/etc/pam.d/), restore the copy: cp /etc/pam.d/common-auth.bak-<date> /etc/pam.d/common-auth
6. Clean exit — do not use reboot, there is no systemd:

   sync
   exec /sbin/init
   

---

CASE 3 — The (initramfs) prompt

It usually comes from a change to crypttab, LVM or the disks without regenerating the initramfs.

1. Check whether the kernel sees the disk: cat /proc/partitions
   • The disk does not appear → a hardware or driver problem. Escalate; check the VM's storage configuration.
   • It appears → carry on.
2. Check whether the tools are present: which cryptsetup lvm
   • No output → they are missing from the initramfs. Go to step 4.
3. Try mounting by hand to confirm the data is intact:

   lvm vgchange -ay
   mkdir /rootfs && mount -o ro /dev/mapper/<volume> /rootfs && ls /rootfs
   

4. Quick recovery: reboot, and in the GRUB menu go into Advanced options for Ubuntu and choose the previous kernel. If it boots:

   sudo update-initramfs -u -k all
   sudo lsinitramfs /boot/initrd.img-$(uname -r) | grep -c sbin/cryptsetup   # must give 1
   

5. If the previous kernel will not boot either: go to Case 4 (chroot) and run the step 4 commands from there.

---

CASE 4 — No GRUB, or none of the above works: the rescue chroot

A universal procedure. It requires the Ubuntu 24.04 ISO mounted in the VM and booting in Try Ubuntu mode.

1. Identify the partitions: lsblk -f srv-tramontana reference: sda1 = ESP (FAT32), sda2 = /boot, sda3 = root.
2. Mount, in this order:

   sudo mount /dev/sda3 /mnt
   sudo mount /dev/sda2 /mnt/boot
   sudo mount /dev/sda1 /mnt/boot/efi
   for d in /dev /dev/pts /proc /sys /sys/firmware/efi/efivars /run; do
       sudo mount --bind "$d" "/mnt$d"
   done
   

   ⚠️ efivars is not optional. Without it, grub-install fails under UEFI with an unclear error.
3. Enter: sudo chroot /mnt /bin/bash
4. Repair whatever applies:

   grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=ubuntu --recheck
   update-grub
   update-initramfs -u -k all
   efibootmgr -v | grep -i ubuntu      # the entry must appear
   

5. Clean exit, unmounting in reverse order:

   exit
   for d in /run /sys/firmware/efi/efivars /sys /proc /dev/pts /dev; do
       sudo umount "/mnt$d"
   done
   sudo umount /mnt/boot/efi /mnt/boot /mnt
   sudo reboot
   

---

CASE 5 — It boots, but takes far longer than usual

This is not an emergency. Normal reference: ~12 seconds.

1. systemd-analyze → separates the kernel from user space.
2. systemd-analyze critical-chain → this is the one that matters, not blame: a slow unit that starts in parallel does not delay the total.
3. journalctl -b -u <unit> → the specific cause.
4. Fix it with a drop-in in /etc/systemd/system/<unit>.d/override.conf, never by editing the original unit. If you are replacing an ExecStart, remember the empty ExecStart= line first.
5. Reboot and measure again. Record the new time in the baseline.

---

RULES THAT PREVENT THE FIRST FOUR CASES

Before…	Always do
Rebooting after touching fstab	mount -a and check that it gives no error
Rebooting after touching crypttab, LVM or the disks	update-initramfs -u -k all and verify with lsinitramfs | grep cryptsetup
Touching PAM or sshd_config	A second session open; validate with sshd -t; reload, not restart
Any configuration change	A .bak-$(date +%F) copy and diff -u afterwards
Any delicate module	A snapshot of the VM, numbered

Never: GRUB_TIMEOUT=0, GRUB_TIMEOUT_STYLE=hidden, deleting kernels by hand, or mounts without nofail other than the root and /boot.

If nothing works. Do not improvise for more than 30 minutes. Rebuild: the agreed RTO is 8 hours, the rebuild runbook is in this same document, and the data is in restic with a tested restore. Tell Operations before you start.

You no longer just boot a server: you know what happens when you do. You know the complete chain — UEFI firmware, the ESP and efibootmgr, GRUB with its real configuration sources, the initramfs and the vicious circle it solves, the kernel and its handover of control to PID 1, and systemd activating default.target — and, above all, you know how to use that chain as a diagnostic tool: the first question when faced with a server that will not boot is always how far did it get. You have intervened in the GRUB menu, you have told rescue from emergency, you have booted with a shell as PID 1 and you have understood why the root is read-only. You have repaired a broken fstab — the scenario 05-04 left announced — you have recovered a root password in two minutes (and with it seen why physical access was left out of the threat model of 06-06), and you have reinstalled GRUB from rescue media with the complete chroot, including the --bind of efivars that makes half of all attempts fail. The runbook now has a decision tree that somebody other than you could follow, which is the definition of a useful procedure.

And along the way you have used chroot to enter somebody else's filesystem and run programs inside it as though it were the root. Remember that: it is the primitive on which everything you will see in 07-05 is built, when you discover that a container is not a small machine.

You know how to look at the boot, but you still do not know how to look inside a running process. In Module 5 you learned to measure with the USE method: vmstat, iostat, free and sar tell you the CPU is at 80%, the disk has high latency or memory is running out. That answers how much, but not what. When health_check.sh starts returning 1 because the application takes 400 ms to respond instead of 40, and the counters say the CPU is idle, the disk quiet and memory plentiful, you will need a different class of tool. In lesson 07-02: Advanced Diagnostics you will learn to ask the system what exactly a process is doing: strace to see its system calls one by one — closing the circle with the layers from Module 1 — perf to profile where it really burns cycles and to read a flame graph, and eBPF with bpftrace and the bpfcc-tools utilities to instrument the kernel in production without strace's prohibitive cost. And you will run head-on into a consequence of your own work: the kernel.yama.ptrace_scope = 1 you applied in 06-06 is going to stop you attaching to your own processes, and understanding why that is a good thing is part of the lesson.