In 03-06 I taught you ps, kill and nohup, and I told you in as many words that none of that is any use for managing a real service. In 03-07 you set up the nightly backup with cron and I warned you that something better existed. This is the lesson where both promises are settled, and it is the heart of the module. Today Tramontana Bookings is started by hand: if srv-tramontana reboots, the application does not come back; if process 1284 dies, nobody brings it up; deploy.sh changes the symbolic link and restarts nothing; and the 04:20 backup depends on a cron line with no real overlap control and no decent logging. By the end you will have tramontana.service written, hardened and enabled, and backup_tramontana.sh turned into a service with its timer.

1. What an init system is and what systemd solves
2. PID 1, cgroups and the unit types
3. systemctl: querying, controlling and enabling
4. Where the units live and what precedence they have
5. Writing a .service unit section by section
6. Hardening the unit and capping it with cgroups v2
7. Targets and runlevels
8. Timers: the replacement for cron
9. Analysing the boot and launching transient jobs
10. Tramontana case: the service, the timer and the complete deployment

The kernel boots, mounts the root filesystem and runs one process: init, with PID 1, from which everything else descends. Its job is to bring the system up, keep the services alive and shut it down in an orderly fashion.

The practical consequence of cgroups is enormous: if your application launches five children and one is orphaned, systemctl stop kills them all, because the kernel accounts for them in the same control group. With a PID file, that orphan carried on writing while you believed you had stopped the service.

Everything in systemd is a unit, and the file's suffix says what type it is:

You have to read the whole output:

• Loaded: which file it comes from and whether it is enabled (it starts on its own when the machine boots) or disabled. A service can be active and disabled: it works now, but it will not come back after a reboot.
• Active: active (running) for a daemon, active (exited) for a oneshot that finished cleanly, failed with the exit code and the signal if it died.
• Tasks, Memory, CGroup: the cgroup's real consumption and the complete process tree, children included, not an estimate. The last lines are journal log, which you will squeeze in 05-06.

This is where the circle closes with 02-06:

enable creates the link inside the .wants of the target named in [Install] and disable deletes it: nothing more. That is why enable does not start the service (for that, enable --now) and disable does not stop it. mask links the unit to /dev/null, so it cannot be started at all — not by hand, not as a dependency, not by a socket — which is what you want during maintenance; unmask reverses it.

sudo systemctl daemon-reload is mandatory after creating or editing any unit file: if you skip it, systemd carries on working with the previous version in memory and you will go mad. And reloading the configuration does not restart the service: that is a separate restart.

edit opens /etc/systemd/system/tramontana.service.d/override.conf, where you write only the section and the directives you want to change. A drop-in is better than editing the original unit: it survives an update of the package that brought it, it makes explicit what you changed and it is removed by deleting one file. A warning about list directives (ExecStart, Environment): to replace them you have to empty them first with a blank ExecStart=, or they accumulate.

In decreasing order of priority: /etc/systemd/system/ (where you write), /run/systemd/system/ (transient units, in RAM) and /lib/systemd/system/ (the packages' ones). Your units go in /etc/systemd/system/; the packages' units are not touched, they are adjusted with drop-ins. A file with the same name in /etc completely replaces the one in /lib.

This is where everybody gets tangled up, because there are two independent axes: the ordering (when it starts relative to another unit) and the dependency (whether it needs that unit to exist).

The classic mistake is putting Requires=postgresql.service and believing it guarantees the database will be ready before you. It does not guarantee it: Requires without After starts both in parallel, so you need both directives. As a rule use Wants=: if the remote logging service fails, you would rather your application started anyway.

A modern service in Go, Java or Python is almost always simple or exec: do not send it to the background with & and do not use nohup, because systemd already takes care of that. The other day-to-day directives:

• Restart=: no (the default), on-failure (a non-zero code or a fatal signal: the right option for an application), always, on-abnormal.
• RestartSec + StartLimitBurst + StartLimitIntervalSec: restarts after 5 s, and if it does so 5 times within 300 s it gives up and stays failed. Without that limit, an application that will not start enters an infinite loop that eats CPU and fills the log; to bring it back, systemctl reset-failed.
• TimeoutStopSec and KillMode=mixed: how long to wait after the SIGTERM before the SIGKILL, and making sure the latter reaches the whole cgroup. It is the "SIGTERM → wait → verify → -9" sequence from 03-06, automated.

It usually has one line, WantedBy=multi-user.target, and it is what enable reads to know which .wants directory to create the link in. A unit with no [Install] cannot be enabled: it will show up as static.

It is one of the most valuable things about systemd, and it is free: a few lines leave your service with far less attack surface than you would achieve by hand.

systemd-analyze security tramontana.service scores the result: the unit we will write at the end of the lesson gets 2.9 OK, against the 9.6 UNSAFE it got before hardening. A practical note: ProtectSystem=strict will break your service the first time, because it writes somewhere you have not declared. Do not disable it: look at the log, add the path to ReadWritePaths and try again. That exercise forces you to know exactly where your application writes.

The resource limits, which systemd applies with cgroups v2, belong to the same [Service] section:

That MemoryMax stops a memory leak taking the database and SSH itself down with it: on exceeding it, the kernel kills inside the service's cgroup and Restart=on-failure brings it back up.

There is also network-online.target, the synchronisation point that says usable networking exists. systemctl get-default says which is the default one, set-default changes it and sudo systemctl isolate rescue.target switches target right now, cutting off the services that are not needed.

A timer is a unit that activates another one: by convention, tramontana-backup.timer fires tramontana-backup.service (the same name, a different suffix).

Always verify the expression before trusting it:

Other ways of firing: OnBootSec=5min (after the machine boots), OnUnitActiveSec=1h (every hour since the last run) and OnStartupSec=. And readable expressions: daily, weekly, Mon..Fri 08:00, *-*-01 03:00 (the 1st of every month).

Persistent=true is the main reason to migrate from cron: if the server was powered off at 04:20, the backup runs as soon as it boots instead of being lost until the following day.

blame sorts by duration; critical-chain shows the chain that really delays the start of one particular unit, which is what matters: a slow service nobody depends on delays nothing. And systemd-run launches a one-off as a transient unit, recorded in the journal and with its limits applied, unlike a nohup.

Decisions worth understanding: Type=exec because the application stays in the foreground and that way a non-existent binary is detected at start-up; User=svc-tramontana with Group=tramontana, the identities from 05-01, and UMask=0027 by the course's convention; ReadWritePaths with only the log and the uploads, because everything else — /opt/tramontana included — must be immutable as far as the application is concerned; and an empty CapabilityBoundingSet= because it listens on 8080 (if it moved to 80 you would add AmbientCapabilities=CAP_NET_BIND_SERVICE, replacing the setcap from 05-02, which was lost on every deployment).

Type=oneshot because the script finishes, and no [Install] because it is not enabled: the timer fires it, and that is why it will show up as static. RequiresMountsFor= is this unit's jewel: if the LVM volume from 05-04 is not mounted, the backup does not run instead of merrily writing into the empty directory on /, a silent failure you would only discover on the day of the restore. Nice=10 and IOSchedulingClass=idle make the backup give way to the application, something that will turn out to matter in 05-07.

inactive (dead) after a oneshot is success: the script finished with code 0; a failed would show the exit code you defined in 04-03. And that final sed withdraws the cron line, which is the half of the job people forget: leaving both things active means two simultaneous backups fighting over the flock every night.

The debt from Module 4: after the atomic ln -sfn, the script checked the health against a process still running the previous release, because the binary was already loaded in memory. Now:

The sudo systemctl restart works without an interactive password thanks to the /etc/sudoers.d/tramontana rule you wrote in 05-02: this is where the three lessons fit together. And systemctl is-active --quiet in a loop avoids the classic mistake of checking the health before the process has opened the port.

• Forgetting daemon-reload. You edit the unit, restart and nothing changes: if you have touched a unit file, daemon-reload before anything else. And enable does not start the service, it only creates the link: use enable --now.
• Requires= without After=. It guarantees no ordering at all: both units start in parallel. Prefer Wants= unless you genuinely cannot function without the other one. And when you migrate to a timer, comment out the cron line in the same change or you will have two simultaneous runs.
• Sending the process to the background with & or nohup in ExecStart. systemd loses sight of it and considers it dead: let it run in the foreground.
• Restart=always without StartLimitBurst. An application that will not start enters a loop that eats CPU and fills the disk with logs. And do not give in to ProtectSystem=strict: the failure tells you where your application writes, so add the path to ReadWritePaths instead of removing the protection.
• Editing a package's unit in /lib/systemd/system. The next update carries off your work: use a drop-in in /etc/systemd/system/<unit>.d/.
• Tip: keep your units in git alongside the scripts. A unit is production code and is reviewed as such.

1. Diagnosing a service that will not start. tramontana.service is failed. List, in order, the five commands you would run to find the cause, and say what each one tells you.
2. A timer with recovery. Write the .service + .timer pair that runs /home/operator/scripts/purge_releases.sh every Monday at 05:10, recovers if the server was powered off, does not overlap with the backup and is logged. Verify the calendar expression.
3. A drop-in instead of an edit. The supplier asks that Tramontana start with TRAMONTANA_MAX_THREADS=8 and a 768 MiB memory limit, without modifying tramontana.service. Do it and demonstrate that it has taken effect.

1.

The fourth command launches the application by hand as its user: is it the app that is failing, or the unit?

It is the step that saves the most time. If it works by hand and not through systemd, the culprit is usually the hardening — a path missing from ReadWritePaths — or the environment, because the unit does not inherit your variables.

2.

Conflicts= plus After= guarantee they do not coincide, the logging is automatic in the journal and there is no need for flock: systemd does not relaunch a unit that is already active.

3. With sudo systemctl edit tramontana.service, which creates the drop-in:

systemctl show gives the effective value (in bytes) and systemctl cat would show the drop-in at the end of the unit without the original having been touched. And Environment= is cumulative: the new variable is added to the ones the unit already defined.

srv-tramontana now governs itself. You know what an init is and what systemd contributes over SysV — parallel dependency-driven boot, real supervision, cgroups that do not lie, structured logging and isolation —; you know the unit types and you handle systemctl from end to end: you read a status line by line, you control with start/restart/reload, you tell enabled from active and disable from mask, you know that enable is literally creating a symbolic link and that daemon-reload is mandatory after touching any unit file. You adjust with drop-ins instead of editing other people's units.

You write a unit from scratch and you understand the two things everybody trips over: the difference between ordering (After/Before) and dependency (Wants/Requires/BindsTo), and which Type= matches each way of starting. You harden with NoNewPrivileges, ProtectSystem=strict and ReadWritePaths, you check it with systemd-analyze security, you cap it with MemoryMax, CPUQuota and TasksMax while watching it with systemd-cgtop, you handle targets, you analyse the boot with blame and critical-chain and you launch transient jobs with systemd-run. And the debts are settled: tramontana.service exists, hardened, with automatic restart and limits, and it starts on its own after a server reboot; backup_tramontana.sh is now tramontana-backup.service fired by a timer at 04:20 with Persistent=true and RequiresMountsFor on the volume from 05-04, with the cron line already withdrawn; and deploy.sh does the systemctl restart it was missing, leaning on the sudoers rule from 05-02. What is missing is the system's memory. Your service already writes to the journal, but you do not know how to query it; /var/log/tramontana/access.log is 412 lines long and growing with nobody rotating it, which was the first debt I announced when closing Module 4 and is still unpaid; and when Marta asks "what happened last night at 03:00?", right now you would not be able to answer precisely. In System Logs: journald and syslog you will see the two logging layers that coexist on Ubuntu, you will squeeze journalctl with all its filters, you will make the journal persistent, you will write a correct /etc/logrotate.d/tramontana tested in simulation, you will connect lib/common.sh to the journal via logger, and you will learn what must never end up written in a log.