The previous lesson ended with a clear limitation of the shell: a lone script is not a system. Somebody has to start meteo-api when the machine powers on, wait for /var/lib/meteora to be mounted, restart it if it falls over, run the aggregator every hour even if the server was switched off at 03:00, apply module 5's limits and hardening to them, and collect their logs. That somebody is the service manager: the process with PID 1 that the kernel starts at the end of boot and that does not die until the machine shuts down.

This lesson has two halves. In the first we follow meteo-01 powering on phase by phase, from the moment the firmware takes control until PID 1 appears, and we learn what can be observed and adjusted in each phase —because half of all serious boot incidents are solved by knowing how to pass a kernel parameter from GRUB—. In the second we take systemd apart: its unit model, the complete anatomy of meteo-api's unit, dependencies, the timers that replace cron, and the "my service won't start" playbook, which is one of the things you will do most often in your professional life.

Performance monitoring is the next lesson; here we deal with what runs, when, and under which rules.

1. Booting meteo-01, phase by phase
2. What problem systemd solved
3. The unit model
4. Anatomy of a unit file: meteo-api in full
5. Service types and how to choose well
6. Dependency versus ordering
7. Targets and runlevels
8. Day-to-day work with systemctl
9. Restart policies and the loop that hides a failure
10. Timers versus cron
11. Socket and path activation
12. User units and loginctl
13. Boot analysis and debugging
14. Graceful shutdown: SIGTERM, SIGKILL and TimeoutStopSec

Phase 1: firmware. When you press the button, the CPU starts executing code from the board's firmware. There are two worlds:

meteo-01 boots via UEFI. Its ESP is mounted at /boot/efi and contains EFI/debian/grubx64.efi and the shimx64.efi signed by Microsoft that makes Secure Boot possible: the firmware verifies shim's signature, shim verifies GRUB's, GRUB verifies the kernel's and the kernel verifies the modules'. The chain exists to prevent a bootkit, malicious code that runs before the operating system and that therefore no antivirus inside the system can see. Its practical price is that a hand-compiled module (a proprietary driver, say) will not load unless you sign it.

What they prove. The kernel only creates /sys/firmware/efi if it booted via UEFI: that is the canonical check. efibootmgr reads the NVRAM variables where the firmware stores the boot order, and lets you add or reorder entries without going into the setup screen. mokutil reports the state of Secure Boot.

Phase 2: boot loader. GRUB reads /boot/grub/grub.cfg —a generated file, never edited by hand: it is produced by update-grub from /etc/default/grub and /etc/grub.d/— and presents the menu. Each entry names a kernel, an initrd and a parameter line:

By pressing e on the entry you can edit that line for a single boot, without touching the disk. It is the most important rescue tool in existence:

Phase 3: kernel and initramfs. GRUB loads the compressed kernel and the initrd into memory and jumps to the kernel, which decompresses itself, initializes module 2's memory management and mounts the initramfs as a temporary root in RAM.

Why does the initramfs exist? Because of a chicken-and-egg problem. meteo-01's real root is on /dev/md0, a RAID 1 over ext4. To mount it you need the raid1 module and the ext4 one… which live inside that very root. The initramfs breaks the circle: it is a compressed cpio with the essential modules and a minimal /init that loads the drivers, assembles the RAID, opens LUKS if there is any, activates the LVM volumes and only then mounts the real root. Afterwards it runs switch_root, which replaces the temporary root with the real one, frees the initramfs's RAM and does an execve of /sbin/init —a link to /lib/systemd/systemd— with PID 1.

What they do and why they matter. If you add a disk or change the storage layout and forget update-initramfs, the machine will boot as far as the initramfs and sit there at an (initramfs) prompt because it cannot find the root: it is one of the most common ways to brick a boot. And dmesg is the kernel's ring buffer, holding everything that has happened since the very first instruction: hardware detection, disk errors, the OOM killer from 02-04, the RAID messages. -T turns the relative timestamps into readable dates.

Phase 4: PID 1. PID 1 is special for three reasons that come from 02-01: the default signal dispositions do not apply to it (the kernel ignores SIGTERM and SIGKILL sent to it, so nobody can kill it by accident), it adopts orphans and reaps their exit statuses, preventing zombies, and if it dies, the kernel panics. Everything else in the system descends from it.

Before systemd, boot was governed by SysV init: shell scripts in /etc/init.d/ that were run in alphabetical order from /etc/rc3.d/ (S01, S02, S03…), one after another, each starting its daemon with a start-stop-daemon. The model had five serious problems:

That last point is the most underrated one and connects directly to 06-02. A SysV script tracked its daemon through a .pid file; if the process daemonized badly, changed PID or spawned children, stop killed the wrong PID and left orphaned processes consuming resources. systemd puts every service in its own control group, so membership is a kernel property, not a guess: stopping the service means signaling the whole cgroup, and the service's CPU, memory and I/O accounting is exact.

What this proves. The CGroup field of status lists every process belonging to the service, including any it spawned itself. No process escapes systemd, because the cgroup is assigned by the kernel at fork time and is inherited.

Everything in systemd is a unit: a text file in INI format describing something that is managed. The types you need to know:

Where they live, in increasing order of priority:

A file in /etc/systemd/system/meteo-api.service completely replaces the package's own. That is almost never what you want: if the package improves its unit, you are stuck with the old one. The right approach is a drop-in, a fragment that is merged with the original:

What they do. systemctl edit opens an editor on /etc/systemd/system/meteo-api.service.d/override.conf, and when you save it runs daemon-reload automatically. systemctl cat is the command you should always run before touching anything: it shows the source file and, below it, each drop-in with its path, so you see the effective configuration and where each directive comes from.

One trap you need to know: in a drop-in, directives that accept lists (such as ExecStart or Environment) accumulate. To replace an ExecStart you must first empty it with a bare ExecStart= line and then set the new one.

Section by section:

• [Unit] describes the unit and its relationships. Description is what shows up in systemctl status and in the logs: write it for a half-asleep human. After= sets ordering, not dependency. RequiresMountsFor= is the correct way to say "do not start if these paths are not mounted": systemd works out the .mount units involved on its own, and it is more robust than naming them by hand, because a mount unit's name is encoded in a peculiar way (/var/lib/meteora → var-lib-meteora.mount).
• [Service] defines how it runs. ExecStartPre runs first and, if it fails, the service does not start: here we reuse the verification script from 07-01 as a preflight check. ExecStart is the main process, always with an absolute path (systemd does not use your shell's PATH). ExecReload reloads the configuration without dropping connections; $MAINPID is one of the few variables systemd expands.
• The hardening block is module 5 turned into directives. NoNewPrivileges sets the kernel bit that prevents gaining privileges through setuid, so not even a compromised setuid binary would serve as a stepping stone. AmbientCapabilities=CAP_NET_BIND_SERVICE is what allows listening on port 443 without being root, and CapabilityBoundingSet sets the ceiling: even if the process wanted another capability, the kernel would not grant it. ProtectSystem=strict mounts the entire file system read-only for the service, and ReadWritePaths opens the three necessary exceptions. PrivateTmp gives it its own /tmp in a mount namespace, which eliminates predictable-temporary-file attacks at the root. SystemCallFilter=@system-service applies a seccomp filter that lets a service's usual set through and blocks the rest by returning EPERM. UMask=0027 sets the creation permissions agreed for Meteora.
• [Install] is used only when you enable. WantedBy=multi-user.target means "when enabled, create a link in multi-user.target.wants/". Without an [Install] section, a service cannot be enabled: you can start it by hand, but it will never start on its own. It is a frequent oversight.

Always verify what you have written before you trust it:

What they add. verify catches misspelled directives and references to units that do not exist, which would otherwise only surface at start time. security scores from 0 (armored) to 10 (exposed) by evaluating dozens of isolation directives, and lists one by one the ones that are missing: it is an automatically generated hardening to-do list.

And the ingestor's unit, simpler because it does not listen on a privileged port:

What is new here. RuntimeDirectory=meteora makes systemd create /run/meteora with the given owner and mode at start time and delete it on stop: it is the correct way to manage the directory where the FIFO /run/meteora/readings.fifo lives, instead of creating it by hand in a script. Nice=-5 gives the ingestor a bit more CPU priority (02-02) because losing incoming readings is unrecoverable, whereas a slow API query is merely annoying.

Type= tells systemd when to consider the service started, which governs when the units that depend on it may start.

The typical mistake, and it is so common that it deserves detail: declaring Type=simple for a program that daemonizes (it forks, the parent exits, the child carries on). systemd sees the process it launched exit immediately and, depending on the restart policy, either takes it for dead and restarts it in a loop, or takes it as started and healthy while the real daemon runs unsupervised. The symptom is a status that says active (exited) or a restart cycle for a service that actually works. The fix is not to wrestle with forking: it is to tell the program not to daemonize (nearly all of them have --foreground or -D) and use Type=exec.

We chose Type=notify for meteo-api for a concrete reason: the service takes about 8 seconds to load the index and warm the cache in /dev/shm/meteora-cache. With simple, systemd would consider it ready instantly and any unit depending on it —or the load balancer watching its state— would act on a service that is still returning errors. With notify, the program calls sd_notify(0, "READY=1") when it can genuinely serve, and systemd waits. WatchdogSec=30 adds the opposite: the service must send WATCHDOG=1 every 30 seconds or systemd considers it hung and restarts it, which catches internal deadlocks that do not kill the process.

This is the distinction that causes the most confusion, and it is very simple: they are independent axes.

The crucial illustration. If you write only Requires=var-lib-meteora.mount, systemd will start the mount and your service at the same time, in parallel: meteo-api may try to open /var/lib/meteora/readings/2026-08-31.dat before the file system is mounted, and fail with ENOENT. And if you write only After=var-lib-meteora.mount, you order things correctly but do not request the mount: if nothing else activates it, your service starts with no data. You almost always need both, and that is why RequiresMountsFor= exists: it generates both at once.

Another important nuance: on shutdown, the ordering is automatically reversed. After=network-online.target implies that at stop time meteo-api is stopped before the network, which is exactly what you want so that connections can be closed cleanly.

And network-online.target deserves a warning: it does not mean "there is Internet connectivity", but "the network manager declares that it has finished configuring the interfaces". It needs a service such as systemd-networkd-wait-online to be enabled; without it, the target is reached immediately and guarantees nothing. A robust service retries the connection instead of relying on boot ordering.

A target does nothing by itself: it is a synchronization point that groups units together. It replaces SysV's runlevels:

Careful with isolate. It stops every unit that is not part of the destination target: run by mistake in production, it takes the services down. As a preliminary check, systemctl list-units --type=target shows which targets are active right now.

Field by field, because that is where the information is:

enable versus start is the beginner's number one confusion: start starts it now and does not survive a reboot; enable prepares future boots but starts nothing today. And daemon-reload versus reload: the first tells systemd to re-read the units; the second tells the service to re-read its configuration. Editing a unit without daemon-reload is a classic: systemd keeps using the old version and you go mad.

Per-unit logs, with the journalctl you already know from 05-04:

Why this beats a log file. Because systemd is the process's parent and captures its stdout and stderr, anything the service prints is recorded, including the messages of a startup failure that would never make it into its own file. Every entry carries the unit, the PID, the UID and the boot it happened in, which allows precise filtering. -b -1 is the command that saves the investigation after an unexpected reboot.

For meteo-api we use on-failure: if somebody stops it on purpose, it must stay stopped. For the ingestor we use always, because there is no legitimate reason for it to exit.

The real danger is the restart loop that hides the failure. Imagine somebody introduces a syntax error into /etc/meteora/meteora.conf: the service starts, fails in 200 ms, systemd restarts it, it fails again… At 200 ms per round that is 5 attempts a second, which fill the journal, burn CPU and —worst of all— make a monitor that samples every 30 seconds see active sometimes and failed other times, so the alert never quite fires and nobody investigates.

That is what the start limiter and RestartSec are for:

What they achieve. With these values, if the service starts more than 5 times in 300 seconds, systemd stops trying and marks it failed with the reason start-limit-hit. That turns a silent loop into a stable, visible state, which does fire the alert. RestartSec=5s also prevents the hammering. If the failure is transient (a dependency that is slow), 5 retries 5 seconds apart give it enough room. After fixing the cause you must run systemctl reset-failed meteo-api.service to clear the counter before starting again; forgetting it is another common slip.

The aggregator must run every hour. In systemd that means two units: the job and its clock.

Directive by directive, with the reasoning:

• OnCalendar=hourly is equivalent to *-*-* *:00:00. The syntax accepts expressions such as Mon..Fri 06:00, *-*-01 03:30 or daily. Always check it before you trust it: systemd-analyze calendar 'Mon..Fri 06:00' --iterations=3 prints the actual next runs.
• Persistent=true stores the last run's timestamp on disk and, if the machine was switched off at the scheduled time, runs the job as soon as it boots. That is what anacron used to do in the classic world, here in a single line.
• RandomizedDelaySec=180 spreads the start by up to a random 3 minutes. With one machine it makes no difference, but with fifty servers all aggregating at 03:00 sharp it avoids the stampede that saturates the shared storage array. AccuracySec=1min also lets systemd batch wakeups and save power; set it to 1us only if you genuinely need precision.
• IOSchedulingClass=idle and Nice=10 in the service make the aggregator yield to meteo-api, both on disk and on CPU. This is exactly the mitigation that will be applied in the case study of 07-04.
• Type=oneshot with TimeoutStartSec=900: the task finishes, it does not keep running, and if it has not ended after 15 minutes it is considered hung.

An important detail: you enable the .timer, not the .service. If you enabled the service, it would also start on every system boot. And to test the task by hand you start the .service, which is the unit that does the work.

Comparison with cron:

That last point is what justifies knowing crontab: it is still omnipresent. The five fields are minute, hour, day of month, month and day of week (0 3 * * 1 = Mondays at 03:00); crontab -e edits the user's own and crontab -l lists it; in /etc/cron.d/ the files carry an extra field with the user. And one warning that causes incidents: cron runs with a minimal PATH (/usr/bin:/bin) and without your session's variables, which is exactly the 07-01 mistake about shell configuration files. Always use absolute paths.

As a rule: on systems with systemd, use a timer; and even so, keep the flock inside the script, because it also protects against a simultaneous manual run.

Socket activation inverts the usual order: systemd opens the listening endpoint and only starts the service when the first connection arrives.

What it achieves. The socket is created by systemd (with privileges) and handed to the service as an inherited file descriptor, so meteo-api does not even need CAP_NET_BIND_SERVICE. What is more, since the socket exists from boot, connections arriving while the service restarts queue in the backlog instead of being refused: you can deploy without dropping connections. And the units that depend on the service can start as soon as the socket is ready, not when the process is ready, which parallelizes the boot. It is the same mechanism inetd has used since the 1980s, but integrated with the rest of the model.

Path activation watches the file system:

What it achieves. As soon as a file matching the pattern appears, systemd starts process-input.service. It is the correct alternative to the while true; do ls ...; sleep 10; done loop, because it uses the kernel's inotify: zero consumption while nothing happens and an immediate reaction when it does.

Every user with a session has their own systemd instance, which manages units in ~/.config/systemd/user/ without root privileges:

Why it matters. By default, user units die when the last session closes: that is correct for a desktop, but surprising on a server. enable-linger keeps the instance alive. And loginctl is your window onto systemd-logind, the component that manages sessions, seats and the pam_systemd you saw in 05-02. Rule of thumb: infrastructure services, like Meteora's, always go in the system manager; user units are for personal tasks.

How to read them, which is where everybody goes wrong. blame sorts by duration, not by impact: a service that takes 6 seconds but starts in parallel with twenty others delays nothing. critical-chain does show the critical path: @ is the instant the unit started and + is how long it took, so the boot only gets shorter if you reduce what appears in that chain. In the example, meteo-api is indeed on the critical path and its verification ExecStartPre is part of those 6.8 seconds: a conscious decision to trade a fast boot for a safe one.

The "my service won't start" playbook, in order:

1. systemctl status meteo-api.service -l --no-pager — the reason is almost always in the last few lines. Look at Active: (is it failed, activating, inactive?) and at the code: status=203/EXEC means "the binary could not be executed" (misspelled path, no execute permission, invalid shebang); status=200/CHDIR means a non-existent WorkingDirectory; status=1 is the program itself exiting with an error.
2. journalctl -u meteo-api.service -n 100 --no-pager — the program's actual message. If it is empty, the failure happened before the program ran.
3. systemctl cat meteo-api.service — check that the effective unit is the one you think it is, drop-ins included. Did you run daemon-reload?
4. systemd-analyze verify /etc/systemd/system/meteo-api.service — syntax and references.
5. Run the command by hand, as the service's user: sudo -u meteora /usr/local/bin/meteo-api --config /etc/meteora/meteora.conf. If it works by hand but not as a service, the difference is in the environment: PATH, working directory, variables, or the hardening.
6. Suspect the hardening, which is the most frequent modern cause. Temporarily comment out ProtectSystem=strict or SystemCallFilter= in a drop-in and try again. If it starts that way, you know which directive to tune; hunt it down with journalctl -k | grep -i seccomp or _AUDIT_FIELD_SYSCALL.
7. Check permissions and SELinux/AppArmor: namei -l /var/lib/meteora/readings walks the whole path showing the owner and mode of each component, and dmesg | grep -i denied gives away the mandatory access control from 05-01.
8. Dependencies: systemctl list-dependencies --failed and systemctl show -p After -p Requires.

When you run systemctl stop meteo-api.service, systemd follows a strict sequence:

1. It sends SIGTERM to the main process (or runs ExecStop if there is one).
2. It waits up to TimeoutStopSec (30 s in our unit).
3. If it has not finished, it sends SIGKILL to every process in the cgroup.

The difference between the two signals is the usual one (03-03): SIGTERM is a polite request the program can catch in order to finish the requests in flight, flush the cache from /dev/shm/meteora-cache, fsync what is pending and close; SIGKILL is carried out by the kernel and cannot be caught, so the process vanishes in the middle of whatever it was doing.

Why the value of TimeoutStopSec genuinely matters. If it is too short, a meteo-api that is finishing writing a block receives SIGKILL halfway through and leaves a truncated readings file —exactly the case detected by the size % 24 != 0 check in the 07-01 script—. If it is too long, a system reboot hangs for minutes waiting on a process that is never going to answer. The correct value is the reasonable worst case for a clean shutdown, plus a margin: for meteo-api, with up to 2 GB of cache to flush, 30 seconds.

On a full shutdown (systemctl poweroff), systemd stops the units in the reverse order of boot, unmounts the file systems, syncs the disk and powers off. If you ever see the message A stop job is running for ... (1min 30s / 2min) during an endless shutdown, you are watching exactly this mechanism: a unit that does not respond to SIGTERM and exhausts its timeout. journalctl -b -1 -p warning will tell you which one it was.

Tips: always use systemctl cat before modifying anything, because it shows the effective configuration; add Documentation= to your units so whoever takes over from you knows where to look; run systemd-analyze security on every new service and treat it as a to-do list; and test reboots for real: a server that has been up for 400 days and has never been rebooted is almost certainly hiding a service that is not enabled.

Turn the check-readings.sh script from 07-01 into a service with a timer that runs every day at 06:15, catches up on the run if the machine was switched off, runs as meteora with no more privileges than necessary, cannot write anywhere, and is considered hung after 10 minutes. Write the two units and the commands to enable and test them.

After a change, meteo-api will not start. systemctl status shows:

List, in order, the checks you would make and what cause each one would confirm.

A colleague defines the ingestor with Restart=always and RestartSec=0, with no start limit. A configuration error makes it fail at startup. Describe what happens on the machine over the following 60 seconds, why the monitoring might not raise an alert, and which three directives you would add.

Solution 1

Key points. ProtectSystem=strict without ReadWritePaths leaves the whole system read-only, which is exactly what the exercise asks for: the checker only reads (its temporary directory is covered by PrivateTmp). An empty CapabilityBoundingSet= removes all capabilities, because a checking script needs none. TimeoutStartSec=600 gives the 10 minutes requested, and since it is Type=oneshot systemd knows the unit has finished when the process exits. SuccessExitStatus=0 1 is the fine detail: our script returns 1 when there are warnings, and we do not want that to mark the unit as failed; a 3 (critical) should. And Persistent=true covers the requirement of catching up on a missed run.

Solution 2

status=203/EXEC means the execve failed: systemd never got to run anything, so the service's journal will be empty and the problem is with the binary or its environment, not with the program. Checks, from most to least likely:

1. ls -l /usr/local/bin/meteo-api — does it exist? Is the path exactly right, with no typo? A deployment that left the binary in /usr/local/sbin produces this error.
2. stat -c '%A %U %G' /usr/local/bin/meteo-api — does it have the execute bit? A botched scp or git checkout can strip it.
3. namei -l /usr/local/bin/meteo-api — can the meteora user traverse every directory in the path? It is enough for one intermediate directory to lose its x bit for this to fail.
4. head -c2 /usr/local/bin/meteo-api and file — if it is a script, does its shebang point to an interpreter that exists? A #!/usr/bin/env python3 without python3 installed produces exactly 203/EXEC.
5. ldd /usr/local/bin/meteo-api | grep 'not found' — is a shared library missing?
6. mount | grep ' /usr/local ' — is it mounted with noexec? It is rare but it happens, above all on hardened separate partitions.
7. systemctl cat meteo-api.service — is there a recent drop-in that changed ExecStart or added a non-existent WorkingDirectory (that would give 200/CHDIR)?
8. Direct test: sudo -u meteora /usr/local/bin/meteo-api --config /etc/meteora/meteora.conf. If it works by hand, review the hardening, starting with SystemCallFilter and ProtectSystem.

Solution 3

What happens. With RestartSec=0 and no limit, the start-fail cycle repeats as fast as the system can fork+execve: if the failure takes 50 ms to show up, that is around 20 runs per second, i.e. roughly 1,200 in the first minute. Consequences: continuous CPU consumption in process creation, thousands of journal entries that can hit journald's rate limits (RateLimitBurst) and cause messages to be dropped —losing precisely the ones that explain the failure—, and I/O pressure on /var/log.

Why the monitoring might not raise an alert. A check that runs systemctl is-active ingestor every 30 seconds has a high probability of landing right in a window where the service is activating or active, because the state oscillates dozens of times a second. The result is an intermittent, flapping alert that many systems suppress as noisy, or simply a series of successful checks. The service never reaches a stable failed state, which is what alerts know how to detect.

The three directives:

With this, the worst case is 5 retries in 25 seconds and then a stable failed state, which fires the alert on the very first check and leaves the journal readable. I would also add Restart=on-failure instead of always if there were any legitimate reason to stop the service, plus a specific alert on the output of systemctl list-units --failed, which is the cheapest and most reliable signal that something is wrong on the machine.

You have followed meteo-01 powering on end to end: the UEFI firmware reading the ESP and verifying signatures with Secure Boot; GRUB presenting the menu whose parameter line is the most valuable rescue tool you have; the kernel and the initramfs, which exists to break the circle of needing the RAID and ext4 modules that live inside the root it cannot yet mount; the switch_root and, finally, PID 1, which cannot be killed, adopts orphans and whose death causes a panic.

And you have taken systemd apart, which solved five real shortcomings of SysV's sequential scripts: parallelism, declared dependencies, on-demand activation, supervision and —the most underrated— one cgroup per service, which turns "the processes of this service" into a kernel fact rather than the guess of a .pid file. On that model you have written meteo-api's complete unit, where every block has its reason: Type=notify because it takes 8 seconds to be genuinely ready, AmbientCapabilities=CAP_NET_BIND_SERVICE to listen on 443 without being root, ProtectSystem=strict with three ReadWritePaths, a seccomp filter, and the cgroup's memory and I/O limits.

And you have seen the mechanisms used every day: the distinction between dependency and ordering, which explains half of all broken boots; enable versus start and daemon-reload versus reload; status read field by field; the restart loop that hides a failure and the three directives that turn it into a visible state; timers with Persistent=true and RandomizedDelaySec versus cron; socket activation, which lets you restart without dropping connections; critical-chain versus blame; the eight-step playbook for "it won't start"; and the SIGTERM → TimeoutStopSec → SIGKILL sequence, whose badly chosen value produces exactly the truncated files we were detecting in the previous lesson.

With this, meteo-01 starts on its own, recovers on its own and runs its tasks on schedule. But a system that boots correctly can still work badly. The service is active (running), the timer fires punctually, no unit is failed… and yet users complain that the API is slow. None of this lesson's tools answers "why?": for that you need a method and some numbers.

That is the next topic: Performance Monitoring and Troubleshooting, where you will learn what to look at in the first 60 seconds of an incident and how to get from a vague symptom to a concrete cause.