The previous five lessons have added security measures one at a time: a controlled network, SSH with keys, a firewall, intrusion detection, encrypted secrets and TLS. Each one solved a specific problem. This lesson does something different: it turns them into a security posture, which is a coherent set of decisions justified by an explicit threat model, with a written record of what is protected, what has been consciously accepted and what falls outside an administrator's remit.

The difference is not rhetorical. With no threat model, hardening degenerates into cargo cult: recipes found in guides get applied because they "sound secure", things break without anybody understanding why, and you end up with a fragile system and a false sense of control. With a threat model, every measure has a reason and every omission is a conscious decision.

You also close the third outstanding incident here — the services still using the vulnerable libssl loaded in memory — and settle the debt you left in 05-01: PAM, the subsystem that really decides how somebody authenticates on this system.

Warning up front. Two of the sections in this lesson — PAM and secure mounts — can leave you unable to log in to your own server if you get them wrong. In both cases the safe procedure is given. Do not skip it: the lab exists precisely so that you learn this where mistakes cost nothing.

Contents

  1. The five principles that order everything so far
  2. Tramontana's threat model
  3. Reducing the attack surface
  4. Hardening accounts and the password policy
  5. PAM: how this system really authenticates
  6. Mandatory access control with AppArmor
  7. Kernel hardening with sysctl
  8. Secure mounts
  9. Vulnerability management and closing the openssl incident
  10. systemd hardening revisited
  11. Backups as the last control against ransomware
  12. A hardening checklist and maintaining the posture

The five principles that order everything so far

Everything you have done in this module answers, without our having said so, to five principles. Naming them lets you apply them to new situations instead of repeating recipes:

Principle What it demands Where you have already applied it
Least privilege Every process and every person with exactly the permissions they need, not one more svc-tramontana with no shell, sudoers with specific commands, an empty CapabilityBoundingSet=
Minimal attack surface What does not exist cannot be attacked A server with no desktop, port 8080 closed to the outside, an allowlist policy in ufw
Defence in depth Several independent layers; one of them failing must not be enough Firewall + hardened SSH + AIDE + auditd + encrypted secrets
Fail safe When something breaks, it must end up closed, not open policy drop by default in nftables, set -euo pipefail in the scripts
Secure by default The initial state must be the restrictive one umask 027, 600 permissions on the secrets, PasswordAuthentication no

The hardest to apply in practice is the second, because it requires taking things away, and taking things away is frightening. The most forgotten is the fourth: a lot of "secure" configuration fails open, and that is worse than not having it, because nobody will notice.

Tramontana's threat model

A threat model answers three questions. Half a page is enough, and it is the half page that makes everything else make sense.

What are we protecting? Two things, in this order:

  1. The guests' personal data (names, dates of stay, amounts in bookings.csv and in the database). Its compromise is harm to third parties, has legal consequences under the GDPR and is irreversible: once leaked, it cannot be "recovered".
  2. The availability of the booking service. Its interruption has a direct economic cost, bounded in time.

From whom? Four realistic actors, ordered by likelihood:

Threat Likelihood Capability The controls we set against it
Automated scanning from the Internet Constant Low: it exploits the known and unpatched Firewall, fail2ban, security updates, SSH without passwords
A leaked or reused credential Medium High: it gets in as a legitimate user Keys instead of passwords, encrypted secrets, rotation, auditd
Your own configuration mistake High Variable, sometimes total --dry-run, copies before editing, netplan try, mount -a, AIDE
Abuse of legitimate access Low High within their remit Least privilege in sudoers, ACLs, access auditing

Look at the third row. The most likely actor is you, and that observation explains why half a dozen of the course's conventions — copy before editing, verify with diff -u, run a dry run, netplan try with automatic rollback, mount -a before rebooting — are security controls in their own right, and not stylistic fussiness.

What do we assume is out of scope? Saying it is as important as the above, because it delimits what is not being protected:

  • A state-resourced attacker or a compromised supply chain. That is beyond any measure an administrator can take on a small company's server.
  • Physical access to the machine. LUKS encryption mitigates disk theft, but an attacker with prolonged physical access and the machine powered on owns the system.
  • A vulnerability in the application itself (SQL injection, business logic). That is development's responsibility, not administration's, and system hardening only limits the subsequent damage.
  • A compromise of the cloud provider or the hypervisor.

And now the underlying criterion: with this model, investing in a NIDS adds little (you already reasoned that in 06-04) and investing in change detection and authentication control adds a great deal. That is making decisions, not following a list.

Reducing the attack surface

What does not exist cannot be attacked, needs no patches and appears in no CVE. It is the principle with the best effort-to-result ratio.

What is listening

$ sudo ss -tulpn
Netid State  Local Address:Port  Peer Address:Port Process
udp   UNCONN 127.0.0.54:53       0.0.0.0:*         users:(("systemd-resolve",pid=712,fd=17))
udp   UNCONN 127.0.0.53%lo:53    0.0.0.0:*         users:(("systemd-resolve",pid=712,fd=13))
tcp   LISTEN 127.0.0.53%lo:53    0.0.0.0:*         users:(("systemd-resolve",pid=712,fd=14))
tcp   LISTEN 0.0.0.0:22          0.0.0.0:*         users:(("sshd",pid=894,fd=3))
tcp   LISTEN 0.0.0.0:8080        0.0.0.0:*         users:(("tramontana",pid=4102,fd=6))
tcp   LISTEN 10.0.2.15:5432      0.0.0.0:*         users:(("postgres",pid=1105,fd=5))

Read each line asking "does it have to be there, and on that address?":

  • systemd-resolve on 127.0.0.53 and 127.0.0.54: local, correct.
  • sshd on 0.0.0.0:22: necessary, and already hardened.
  • tramontana on 0.0.0.0:8080: listening on every interface. The firewall blocks it from outside, but that is defence in a single layer. The right thing is for the application to listen only on 127.0.0.1, because when the reverse proxy from 08-01 arrives it will be the one talking to it, and that way the firewall becomes the second line instead of the only one.
  • postgres on 10.0.2.15:5432: already restricted to the internal interface. Correct.

The first fix, then:

$ sudo cp -p /etc/tramontana/app.conf /etc/tramontana/app.conf.bak-$(date +%F)
$ sudo chattr -i /etc/tramontana/app.conf
$ echo 'listen=127.0.0.1' | sudo tee -a /etc/tramontana/app.conf
$ sudo chattr +i /etc/tramontana/app.conf
$ sudo systemctl restart tramontana.service
$ sudo ss -tulpn | grep 8080
tcp   LISTEN 127.0.0.1:8080  0.0.0.0:*  users:(("tramontana",pid=4318,fd=6))

That is defence in depth applied: it now takes two failures — a badly written firewall rule and a listener that is too broad — to expose 8080. Before, one was enough.

What is enabled

$ systemctl list-unit-files --state=enabled --type=service --no-pager
UNIT FILE                    STATE   PRESET
apparmor.service             enabled enabled
auditd.service               enabled enabled
cron.service                 enabled enabled
fail2ban.service             enabled enabled
ModemManager.service         enabled enabled
multipathd.service           enabled enabled
postgresql.service           enabled enabled
ssh.service                  enabled enabled
tramontana.service           enabled enabled
unattended-upgrades.service  enabled enabled

ModemManager manages modems and mobile broadband. On a server virtual machine it makes no sense at all. And here it is worth understanding the difference between the two ways of switching something off, because they are not equivalent:

Operation What it does Can it come back on its own
disable Deletes the [Install] links; it does not start at boot Yes: if another unit declares it in Wants=
mask Links the unit to /dev/null; it cannot be started No: neither by hand nor through a dependency
$ sudo systemctl disable --now ModemManager.service
$ sudo systemctl mask ModemManager.service
Created symlink /etc/systemd/system/ModemManager.service → /dev/null.

mask for what must never start; disable for what you might want to start by hand one day. And the next step, when you are sure, is to uninstall the package: a binary that is not on the disk has no vulnerabilities.

$ sudo apt purge modemmanager
$ sudo apt autoremove --purge

On the desktop, to close the section: a graphical environment adds hundreds of packages, an X or Wayland server, a session manager and a browser — each with its own surface and its own patching calendar. srv-tramontana not having one is not austerity, it is the biggest surface reduction you can make on a server, and that is why the Ubuntu Server installer does not offer it by default.

Hardening accounts and the password policy

Three audits worth keeping in a script, because they catch serious mistakes and are one line each:

# 1. Accounts with UID 0: there must be only one, root
$ awk -F: '($3 == 0) {print $1}' /etc/passwd
root

# 2. Accounts with no password (empty second field in shadow): serious
$ sudo awk -F: '($2 == "") {print $1 " HAS NO PASSWORD"}' /etc/shadow

# 3. System accounts (UID < 1000) with a valid login shell
$ awk -F: '($3 < 1000 && $7 !~ /(nologin|false|sync)$/) {print $1, $3, $7}' /etc/passwd
root 0 /bin/bash
sync 4 /bin/sync

All three results are correct: a single UID 0, no account without a password, and the only two system accounts with a shell are root — needed for recovery — and sync, which is a harmless historical case. The fact that svc-tramontana does not appear in the third list confirms that the -s /usr/sbin/nologin from 05-01 did its job.

The default policy lives in /etc/login.defs, and it affects the accounts created from now on:

# /etc/login.defs (modified values)
PASS_MAX_DAYS   365     # maximum age
PASS_MIN_DAYS   1       # stops it being changed several times in a row to get back to the old one
PASS_WARN_AGE   14      # days of warning before it expires
UMASK           027     # consistent with the course's convention
SHA_CRYPT_MIN_ROUNDS 5000
ENCRYPT_METHOD  YESCRYPT

PASS_MIN_DAYS 1 is less obvious than the rest: without it, somebody who is required to change their password can change it five times in a row to exhaust the history and go back to the original. It is an old trick and it still works wherever this is not set.

And for the accounts that already exist, chage:

$ sudo chage -M 365 -m 1 -W 14 luis
$ sudo chage -l luis
Last password change                                    : Aug 04, 2026
Password expires                                        : Aug 04, 2027
Password inactive                                       : never
Account expires                                         : never
Minimum number of days between password change           : 1
Maximum number of days between password change           : 365
Number of days of warning before password expires        : 14

# The intern's account has an end-of-placement date: it expires automatically
$ sudo chage -E 2026-12-31 intern
$ sudo chage -l intern | grep 'Account expires'
Account expires                                         : Dec 31, 2026

chage -E deserves attention: it is the correct way to manage a temporary account. It expires on its own on the expected date, without depending on somebody remembering to close it — and forgotten leavers are one of the most common routes to compromise there is.

PAM: how this system really authenticates

In 05-01 we deferred PAM. Now is the moment, because it is the piece that really decides who gets into this system and on what conditions.

PAM (Pluggable Authentication Modules) is a layer of indirection: the programs that need to authenticate — login, sshd, sudo, su, passwd — do not implement the logic themselves, they ask PAM. That makes it possible to change the policy (demand strong passwords, lock out after failures, add a second factor) without modifying or recompiling any program. It is the reason it exists.

The structure

Each service has its file in /etc/pam.d/, and there are common files that the rest include:

$ ls /etc/pam.d/ | head -12
common-account
common-auth
common-password
common-session
cron
login
passwd
sshd
su
sudo

Each line has three parts: type, control and module.

The four types, which are four different questions answered at different moments:

Type The question it answers
auth Are they who they say they are? (password, key, second factor)
account Are they allowed in right now? (expired account, time of day, origin)
password Is the new password they want to set acceptable?
session What has to be set up and cleaned up around the session (limits, logging, home)

Confusing auth with account is the most common misunderstanding: a correct password on an expired account passes auth and fails account.

The controls determine what happens according to the module's result:

Control If the module fails If it succeeds
required The stack fails, but the rest still run Continues
requisite It fails and stops right there Continues
sufficient Ignored Immediate success, if no earlier required failed
optional Ignored (unless it is the only one) Continues

required versus requisite has a rationale that is not obvious: required goes on running the stack so as not to reveal at which step it failed. If an attacker could tell "no such user" from "wrong password" by the response time or the message, they would have an oracle for enumerating users.

The modern, more precise syntax uses square brackets: [success=ok default=die] lets you specify the action for each return code. It is what you will see in Ubuntu's files.

The procedure for not locking yourself out

Before you touch a single line. A syntax error in common-auth can prevent every login, the local console included, and then the only way out is recovery mode:

  1. Leave a root session open in another terminal, and do not close it until you have finished:
    $ sudo -i
    # (do not close this session)
    
  2. Copy the file before editing it, following the course's convention:
    $ sudo cp -p /etc/pam.d/common-password /etc/pam.d/common-password.bak-$(date +%F)
    
  3. Test in a third terminal, never in the one holding the root session.
  4. Keep the VM console to hand and know how to enter recovery mode (covered in depth in 07-01).

It is exactly the same discipline as netplan try and the second SSH session: never close the door you are coming in through.

pam_pwquality: demanding decent passwords

$ sudo apt install libpam-pwquality
# /etc/security/pwquality.conf
minlen = 12          # minimum length
minclass = 3         # at least 3 of: lower case, upper case, digits, symbols
maxrepeat = 3        # no more than 3 identical consecutive characters
dictcheck = 1        # rejects dictionary words
usercheck = 1        # rejects passwords containing the username
enforce_for_root = 1 # root too: without this, root is exempt
retry = 3
# /etc/pam.d/common-password (modified line)
password  requisite  pam_pwquality.so retry=3
$ passwd luis
New password: tramontana2026
Bad password: it is based on a dictionary word.
New password: Kj8-mQ2vX9pL
passwd: password updated successfully

Two underlying comments. The first: minlen = 12 with minclass = 3 is more sensible than the "8 characters with an upper-case letter, a number and a symbol" policies that produce Passw0rd! — length contributes far more than forced complexity, and that is what current guidance says (NIST SP 800-63B). The second: enforce_for_root = 1 is easy to forget, and without it the most important account on the system is the only one exempt from the policy.

pam_faillock: locking out after failed attempts

It complements fail2ban, which acts on the IP; faillock acts on the account, so it also covers attempts from the console or from different IPs.

# /etc/security/faillock.conf
deny = 5              # locks out after 5 failures
fail_interval = 900   # occurring within 15 minutes
unlock_time = 600     # automatic unlock after 10 minutes
even_deny_root        # root too
root_unlock_time = 60 # but root unlocks in 1 minute: avoids a total lockout
audit                 # records the attempt
silent                # does not reveal to the attacker that the account is locked
# /etc/pam.d/common-auth
auth  required  pam_faillock.so preauth
auth  [success=1 default=ignore]  pam_unix.so nullok
auth  [default=die]  pam_faillock.so authfail
auth  sufficient  pam_faillock.so authsucc
auth  requisite  pam_deny.so
auth  required  pam_permit.so

The order matters and it is not intuitive: preauth checks whether the account is already locked before asking for credentials, authfail counts the failure, and authsucc clears the counter after a successful login. Missing out authsucc produces a slow but inevitable lockout, because the counter is never reset.

And the combination of even_deny_root with root_unlock_time = 60 is deliberate: protecting root without creating the possibility of a total system lockout.

$ faillock --user luis
luis:
When                Type  Source                                           Valid
2026-08-18 14:22:11 RHOST 203.0.113.44                                         V
2026-08-18 14:22:14 RHOST 203.0.113.44                                         V

$ sudo faillock --user luis --reset     # unlock by hand

pam_limits: resource limits

# /etc/security/limits.conf
# domain       type    item      value
*              hard    nproc     4096      # curbs a process bomb
*              hard    core      0         # do not generate dumps: they can contain secrets
svc-tramontana soft    nofile    8192
svc-tramontana hard    nofile    16384

Two of these lines are security measures rather than performance ones. nproc limits the damage of a loop that creates processes without control — accidental or otherwise. And core 0 avoids memory dumps, which contain everything the process had in RAM at that moment: including the password you went to such lengths to encrypt in 06-05.

Careful with the scope: pam_limits applies to sessions that go through PAM. For a systemd service, the limits go in the unit (LimitNOFILE, LimitNPROC), as you saw in 05-07.

A second factor, briefly

libpam-google-authenticator adds a TOTP code to SSH authentication. It is effective against credential theft, and on a server with key-based access the marginal benefit is smaller than on one with passwords. If it is adopted, there are two things you cannot forget: keeping the recovery codes off the server, and deciding what happens if the TOTP device is lost. Without that, it is an elegant way of locking yourself out.

Closing the finding: the PasswordAuthentication exception

In 06-04 the logs showed that luis logged in with password, even though you disabled it in 06-02. Time to track it down, starting with the authoritative source:

$ sudo sshd -T | grep -i -E 'passwordauth|kbdinteractive'
passwordauthentication no
kbdinteractiveauthentication no

The global configuration is correct. But sshd -T with no arguments shows the global configuration; Match blocks are conditional and do not appear there. You have to ask about the specific case:

$ sudo grep -rn -A3 'Match' /etc/ssh/sshd_config /etc/ssh/sshd_config.d/
/etc/ssh/sshd_config.d/60-tramontana.conf:12:Match User luis
/etc/ssh/sshd_config.d/60-tramontana.conf-13-    PasswordAuthentication yes

$ sudo sshd -T -C user=luis,host=laptop-luis,addr=10.0.2.44 | grep -i passwordauth
passwordauthentication yes

There it is. A Match User luis block with the exception, put in place "temporarily" while Luis generated his key, and never withdrawn. It is the most likely actor in the threat model — your own mistake — in its most characteristic form.

$ sudo cp -p /etc/ssh/sshd_config.d/60-tramontana.conf{,.bak-$(date +%F)}
$ sudo sed -i '/^Match User luis$/,+1d' /etc/ssh/sshd_config.d/60-tramontana.conf
$ sudo sshd -t && echo "syntax OK"
syntax OK
$ sudo sshd -T -C user=luis,host=laptop-luis,addr=10.0.2.44 | grep -i passwordauth
passwordauthentication no
$ sudo systemctl reload ssh

Note sshd -T -C: the -C option evaluates the configuration for a specific context of user, host and address, resolving the Match blocks. It is the only reliable way to know what configuration applies to somebody, and it deserves a place in your memory: half of the baffling SSH problems are a forgotten Match.

Before reloading, the usual discipline: sshd -t validates, the second session is open, and reload is used instead of restart.

Mandatory access control with AppArmor

The permissions you have known since 02-07 are DAC (Discretionary Access Control): discretionary because the owner of a resource decides who gets access. They have a structural limit: a process can do everything its user can do. If Tramontana's application is compromised, the attacker can read anything readable by svc-tramontana and write anywhere that user can write.

MAC (Mandatory Access Control) adds a layer the owner cannot relax: a policy, defined by the administrator and enforced by the kernel, that says what this program can do, regardless of who runs it.

DAC (Unix permissions) MAC (AppArmor / SELinux)
Who decides The file's owner The administrator, in the policy
Unit of control User and group Program (profile)
Can the process relax it Yes, within its user's remit No
What it contains A compromise of the process A compromise of the process and of its user
$ sudo aa-status
apparmor module is loaded.
32 profiles are loaded.
28 profiles are in enforce mode.
   /usr/bin/man
   /usr/sbin/sshd
   ...
2 profiles are in complain mode.
2 processes have profiles defined.

The two modes are the key to the working method:

  • complain: it does not block, it only records what it would have blocked. It is the learning mode.
  • enforce: it blocks. It is the production mode.

And the professional procedure is always the same: write the profile, put it in complain, exercise the application for real, collect the denials and extend the profile, and only then switch to enforce. Starting in enforce means breaking the service and discovering the missing accesses in the worst possible way.

A profile for Tramontana's application

$ sudo apt install apparmor-utils
# /etc/apparmor.d/opt.tramontana.app.tramontana
abi <abi/4.0>,
include <tunables/global>

/opt/tramontana/releases/*/tramontana {
  include <abstractions/base>
  include <abstractions/nameservice>
  include <abstractions/openssl>

  # Its own binary: read and execute
  /opt/tramontana/releases/*/tramontana        mr,
  /opt/tramontana/releases/*/**                r,
  /opt/tramontana/app/**                       r,

  # Configuration: read only. Even if the process is compromised,
  # it cannot modify its own configuration.
  /etc/tramontana/app.conf                     r,

  # Credential delivered by systemd (the service's private tmpfs)
  /run/credentials/tramontana.service/*        r,

  # Its own logs: create and append, never overwrite or delete
  /var/log/tramontana/                         r,
  /var/log/tramontana/*.log                    rw,

  # Data uploaded by the guests: read and write
  /opt/tramontana/shared/uploads/              r,
  /opt/tramontana/shared/uploads/**            rwk,

  # Network: TCP over IPv4/IPv6 only. No raw unix, no raw packets.
  network inet stream,
  network inet6 stream,

  # Nothing else is permitted: AppArmor denies by default.
  # In particular, the following are implicitly DENIED:
  #   /etc/shadow, /home/**, /srv/tramontana/backups/**,
  #   /root/**, executing /bin/sh and loading modules.
}

That last comment is what makes the profile valuable. An attacker running code in the application's context cannot launch a shell, cannot read the backups, cannot touch the home directories and cannot modify its own configuration — things DAC would partly allow them to do, and which are precisely the first steps of any post-exploitation.

# 1. Load in learning mode
$ sudo apparmor_parser -r /etc/apparmor.d/opt.tramontana.app.tramontana
$ sudo aa-complain /opt/tramontana/releases/*/tramontana

# 2. Exercise the application for real: requests, a file upload, a deployment
$ curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/houses
200
$ ~/scripts/health_check.sh; echo "status: $?"
status: 0

# 3. Collect the recorded denials
$ sudo journalctl -k --since "10 min ago" | grep 'apparmor="ALLOWED"'
kernel: audit: type=1400 apparmor="ALLOWED" operation="open"
  profile="/opt/tramontana/releases/*/tramontana" name="/opt/tramontana/shared/templates/invoice.html"
  pid=4318 comm="tramontana" requested_mask="r" denied_mask="r"

In complain mode the marker is ALLOWED with a denied_mask: it means "this would have been blocked". Here a legitimate access the profile did not contemplate has turned up — the templates — so it is added and the cycle repeats:

  /opt/tramontana/shared/templates/**          r,
# 4. Extend it with assistance if you prefer, and switch to enforce
$ sudo aa-logprof
$ sudo aa-enforce /opt/tramontana/releases/*/tramontana
Setting /opt/tramontana/releases/*/tramontana to enforce mode.
$ sudo systemctl restart tramontana.service
$ ~/scripts/health_check.sh; echo "status: $?"
status: 0

From here on, a real denial appears as apparmor="DENIED", and it is the first thing to look at when the service starts failing after a deployment that changes paths:

$ sudo journalctl -k -f | grep apparmor

SELinux, for anybody heading to RHEL:

AppArmor SELinux
Distributions Ubuntu, Debian, SUSE RHEL, Fedora, CentOS Stream
Identifies objects by The file's path A label on the inode
Learning curve Gentle; readable profiles Steep; more powerful
Diagnosis journalctl + aa-logprof ausearch + audit2allow, restorecon
Permissive mode Per profile (complain) Global or per domain

The path-versus-label difference has a practical consequence: in SELinux, moving a file can change its context and break access (hence restorecon); in AppArmor, a symbolic link or a bind mount can bypass a path-based rule. Each model has its weak spot.

Kernel hardening with sysctl

In 06-03 you applied the network sysctl settings. These are the kernel's own, and they are purely about security: the performance ones are covered in 07-03.

# /etc/sysctl.d/60-hardening.conf

# Only root can read the kernel buffer. Stops memory addresses and hardware
# details useful for building an exploit from leaking.
kernel.dmesg_restrict = 1

# Hides kernel pointers in /proc and in the logs.
kernel.kptr_restrict = 2

# Full ASLR: randomises the address space, the heap included.
# 2 is the default in Ubuntu; it is declared here to be explicit.
kernel.randomize_va_space = 2

# Prevents creating hard links to files you do not own and following other
# people's symbolic links in directories with the sticky bit.
# It closes a whole family of race attacks in /tmp.
fs.protected_hardlinks = 1
fs.protected_symlinks = 1

# The same protection for FIFOs and regular files in world-writable
# directories.
fs.protected_fifos = 2
fs.protected_regular = 2

# Only root can use BPF. It greatly reduces that subsystem's surface.
kernel.unprivileged_bpf_disabled = 1

# A process can only debug its direct descendants.
# It stops a compromised process reading the memory of another one belonging to
# the same user (and with it any secrets that other one has loaded).
kernel.yama.ptrace_scope = 1

# Do not let unprivileged users create user namespaces.
# CAREFUL: it breaks unprivileged containers. Commented out because in
# 07-05 you are going to need it; uncomment only on servers with no containers.
#kernel.unprivileged_userns_clone = 0
$ sudo sysctl --system 2>&1 | grep -A9 '60-hardening'
* Applying /etc/sysctl.d/60-hardening.conf ...
kernel.dmesg_restrict = 1
kernel.kptr_restrict = 2
kernel.randomize_va_space = 2
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
fs.protected_fifos = 2
fs.protected_regular = 2
kernel.unprivileged_bpf_disabled = 1
kernel.yama.ptrace_scope = 1

$ sysctl kernel.yama.ptrace_scope
kernel.yama.ptrace_scope = 1

Two warnings to note down, because they are the two ways this can bite you later:

  • kernel.yama.ptrace_scope = 1 can break debuggers. strace or gdb against a process that is already running and is not your child will stop working without sudo. In 07-02 you are going to use exactly those tools; remember that this parameter is the explanation if you get Operation not permitted.
  • kernel.unprivileged_userns_clone = 0 breaks unprivileged containers, and in 07-05 you are going to set up Docker. That is why it is left commented out with the reason written beside it. A hardening parameter with a note explaining why it is not enabled is quality documentation; deleting it and forgetting the reason is not.

And the general criterion: every line in this file carries its comment. A sysctl.d with fifteen values and no explanation is impossible to maintain, because nobody — you included, in six months — will know whether any of them can be removed.

Secure mounts

Three mount options that limit the damage in the directories where anybody can write:

Option Effect
noexec Nothing can be executed from there
nosuid SUID/SGID bits are ignored
nodev Device files are ignored

The use case is direct: an attacker who manages to write a file usually needs to run it, and /tmp is the place where they can almost always write. With noexec, that step fails.

And this is where the lesson insists on something the hardening guides tend to leave out: noexec on /tmp breaks real things. Several installers extract to /tmp and run from there; so do some language package managers. You have to check first:

# 1. Check the impact live, without touching fstab
$ sudo mount -o remount,noexec,nosuid,nodev /tmp

# 2. Exercise what might break
$ sudo apt install --reinstall -y tree >/dev/null && echo "apt: OK"
apt: OK
$ ~/scripts/deploy.sh --dry-run 3.2.1 && echo "deployment: OK"
deployment: OK
$ sudo needrestart -r l >/dev/null && echo "needrestart: OK"
needrestart: OK

# 3. And check that the restriction really does something
$ printf '#!/bin/sh\necho hello\n' > /tmp/test.sh && chmod +x /tmp/test.sh
$ /tmp/test.sh
bash: /tmp/test.sh: Permission denied

That last block is the positive verification: it is not enough that nothing has broken, you have to check that the measure does something. With both checks done, it is made persistent:

# /etc/fstab
UUID=3f8a...  /tmp      ext4  defaults,noatime,noexec,nosuid,nodev  0  2
/tmp          /var/tmp  none  bind,noexec,nosuid,nodev              0  0
tmpfs         /dev/shm  tmpfs defaults,noexec,nosuid,nodev          0  0
$ sudo systemctl daemon-reload
$ sudo mount -a && echo "fstab OK"
fstab OK
$ findmnt -o TARGET,OPTIONS /tmp /var/tmp /dev/shm
TARGET     OPTIONS
/tmp       rw,noexec,nosuid,nodev,relatime
/var/tmp   rw,noexec,nosuid,nodev,relatime
/dev/shm   rw,noexec,nosuid,nodev

mount -a before rebooting is still compulsory, for the reason you have known since 05-04: a badly written fstab stops the server booting.

Vulnerability management and closing the openssl incident

Reading a CVE without fooling yourself

A CVE is the unique identifier of one specific vulnerability. Its CVSS is a score from 0 to 10 accompanied by a vector describing how it is exploited:

CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H  →  9.8 CRITICAL
        │    │    │    │    │    │  └─ Confidentiality/Integrity/Availability: high
        │    │    │    │    │    └─ Scope: unchanged
        │    │    │    │    └─ User interaction: none
        │    │    │    └─ Privileges required: none
        │    │    └─ Attack complexity: low
        │    └─ Attack vector: network

And the point to internalise: the base score is not the real risk to you. A 9.8 in a component you do not have installed, or that is not exposed, or that is already mitigated by another layer, is less urgent than a 6.5 in the service facing the Internet. The vector is more informative than the number: AV:N (exploitable over the network) with no privileges and no user interaction is what forces you to act today.

$ apt list --upgradable
$ pro security-status
1543 packages installed:
     1489 packages from Ubuntu Main/Restricted repository
       54 packages from Ubuntu Universe/Multiverse repository
Ubuntu Pro is not attached. Ubuntu Pro would provide security updates for
54 packages until 2034.

$ sudo apt install debsecan && debsecan --suite noble --format summary | head -5

Closing the incident

On 18 August at 06:12, unattended-upgrades updated openssl and libssl3t64. The package is up to date on disk, but the processes that started earlier still have the old library loaded in memory. It is a vulnerability with paperwork: the report says it is patched and the service is still exposed.

$ grep -E 'openssl|libssl' /var/log/apt/history.log | tail -2
Start-Date: 2026-08-18  06:12:03
Upgrade: libssl3t64:amd64 (3.0.13-0ubuntu3.4, 3.0.13-0ubuntu3.5), openssl:amd64 (3.0.13-0ubuntu3.4, 3.0.13-0ubuntu3.5)

# The direct proof: processes with the OLD library (marked DEL: deleted from
# disk but still mapped in memory)
$ sudo lsof 2>/dev/null | grep -E 'DEL.*libssl'
postgres  1105 postgres  DEL  REG  253,0  /usr/lib/x86_64-linux-gnu/libssl.so.3
sshd       894 root      DEL  REG  253,0  /usr/lib/x86_64-linux-gnu/libssl.so.3

$ sudo apt install needrestart
$ sudo needrestart -r l
Scanning processes...
Scanning candidates...
Scanning linux images...

Services to be restarted:
 systemctl restart [email protected]
 systemctl restart ssh.service

Service restarts being deferred:
 (none)

No containers need to be restarted.
No user sessions are running outdated binaries.
No VM guests are running outdated hypervisor (qemu) binaries on this host.

Two services. The order matters, and for two different reasons:

# 1. The database first, and with the application ready to reconnect.
#    Restarting postgresql cuts tramontana.service's open connections.
$ sudo systemctl restart [email protected]
$ sudo systemctl restart tramontana.service
$ ~/scripts/health_check.sh; echo "status: $?"
status: 0

# 2. SSH last, with RELOAD rather than restart, and with the second session open.
#    reload does not cut existing sessions; restart should not either, but
#    reload is the correct operation and there is no reason to take the risk.
$ sudo sshd -t && sudo systemctl reload ssh

An important nuance about ssh: reload re-reads the configuration, but it does not replace the library loaded in the master process. For that you have to restart the daemon — which does not cut sessions already established, because each one lives in its own child process. With the second session open as a safety net:

$ sudo systemctl restart ssh
$ sudo systemctl is-active ssh
active
# Verify from the second session that you can open a third BEFORE closing anything

And the closing verification:

$ sudo lsof 2>/dev/null | grep -c -E 'DEL.*libssl'
0
$ sudo needrestart -r l
No services need to be restarted.
$ sudo ss -tulpn | grep -E ':22|:5432|:8080'
tcp LISTEN 0.0.0.0:22       users:(("sshd",pid=8841,fd=3))
tcp LISTEN 10.0.2.15:5432   users:(("postgres",pid=8902,fd=5))
tcp LISTEN 127.0.0.1:8080   users:(("tramontana",pid=8977,fd=6))

That 0 is the closure. The second incident is closed, and with it all three.

So that it does not happen again, needrestart is configured in list mode and integrated into the review:

# /etc/needrestart/needrestart.conf
$nrconf{restart} = 'l';       # only list, never restart on its own
$nrconf{kernelhints} = 1;     # warn if the running kernel is not the installed one
# Add to health_check.sh: restarts pending after an update
$ sudo needrestart -r l -p >/dev/null; echo "needrestart exit code: $?"
needrestart exit code: 0

The decision not to restart automatically is deliberate and consistent with the unattended-upgrades of 05-03: an unsupervised restart of the database at six in the morning is a service interruption Marta must know about, not a side effect.

systemd hardening revisited

In 05-05 you hardened tramontana.service. Now it gets measured and improved:

$ systemd-analyze security tramontana.service | tail -12
→ Overall exposure level for tramontana.service: 3.4 MEDIUM 🙂

3.4 MEDIUM for a network service is not bad, and there is room. The missing directives, each with its reason:

# /etc/systemd/system/tramontana.service.d/hardening.conf
[Service]
# The service does not need to tune the kernel or load modules.
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes

# It must not manipulate the cgroup hierarchy.
ProtectControlGroups=yes

# It must not create SUID/SGID files: it closes a classic persistence route.
RestrictSUIDSGID=yes

# Prevents changing the process's "personality" to emulate another architecture.
LockPersonality=yes

# Only system calls typical of a normal service. The rest are rejected
# with EPERM in the kernel, before reaching the process's code.
SystemCallFilter=@system-service
SystemCallFilter=~@privileged @resources @obsolete
SystemCallArchitectures=native

# Only the socket families it needs.
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictNamespaces=yes
RestrictRealtime=yes

# No access to devices or to /home
PrivateDevices=yes
ProtectHome=yes
ProtectProc=invisible
ProcSubset=pid

# CAREFUL: MemoryDenyWriteExecute breaks JIT runtimes
# (Java, Node.js, .NET, some Python). Only if the application is native.
MemoryDenyWriteExecute=yes

SystemCallFilter deserves an explanation because it is the highest-value directive: it installs a seccomp filter in the kernel that rejects system calls outside the permitted set. An exploit that achieves code execution inside the process finds that mount, ptrace, init_module or setuid fail with EPERM — not because of missing user permissions, but because the kernel does not accept them from this process. That is real containment, not cosmetic configuration.

And the warning about MemoryDenyWriteExecute is the kind of detail that separates a useful guide from a copied list: it stops a memory page being writable and executable at the same time, which blocks many exploitation techniques, and it breaks any engine with just-in-time compilation. If Tramontana's application were Java or Node, this line would stop it starting at all.

$ sudo systemctl daemon-reload && sudo systemctl restart tramontana.service

# Verify that the service STILL WORKS: hardening and breaking is not hardening
$ ~/scripts/health_check.sh; echo "status: $?"
status: 0
$ curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/houses
200

$ systemd-analyze security tramontana.service | tail -3
→ Overall exposure level for tramontana.service: 1.6 OK 🙂

From 3.4 MEDIUM to 1.6 OK, with the service running and verified. The order — apply, verify, measure — is the same as in credential rotation and in performance diagnosis. Applying without verifying is how you discover at three in the morning that the hardening took the service down.

Backups as the last control against ransomware

When everything above fails, the backups remain. And against ransomware there is a detail that is often overlooked: a permanently mounted backup is reachable by the ransomware, because the process encrypting the system's files also encrypts those on the mounted volume. A backup encrypted by the attacker is not a backup.

The three properties that make a backup resistant:

Property How it is achieved Status at Tramontana
Offline or immutable Disconnected media, or storage with enforced retention Outstanding for the 3-2-1 off-site destination
A separate credential The repository password is not reachable from the compromised server Partial: it is in pass, but the server can read it
Tested restore A documented periodic test Done: the three scenarios of 05-08

The concrete improvement that follows: restic supports append-only mode on the remote destination, in which the client can write new backups but cannot delete existing ones. With that, an attacker with access to the server cannot destroy the history. It is the measure to ask the external storage provider for, and it goes on the checklist.

A hardening checklist and maintaining the posture

This is the table that gets handed over, aligned with the structure of the CIS Benchmarks and with what you have actually done in the course. The three columns are deliberate: without the evidence column, a checklist is a statement of intent.

# Control Status Evidence
1 Filesystem: noexec,nosuid,nodev on /tmp, /var/tmp, /dev/shm Done findmnt -o TARGET,OPTIONS
2 Encryption at rest of the backup volume Done cryptsetup luksDump, /etc/crypttab
3 Signed repositories; official updates only Done /etc/apt/sources.list.d/*.sources, /etc/apt/keyrings/
4 Automatic security updates Done unattended-upgrades, -security only
5 Services pending restart after an update Done needrestart -r l with nothing pending; incident closed
6 Minimal listening surface Done ss -tulpn: 22, 5432 internal, 8080 on localhost
7 Unnecessary services disabled and masked Done ModemManager masked and purged
8 No graphical environment Done An Ubuntu Server installation
9 Firewall with a default-deny policy Done ufw status verbose; documented nftables ruleset
10 Network sysctl hardened Done /etc/sysctl.d/ (06-03)
11 Kernel sysctl hardened Done /etc/sysctl.d/60-hardening.conf, commented
12 SSH: no root, no passwords, keys only Done sshd -T -C user=...; the Match exception removed
13 Lockout on failed attempts (IP and account) Done fail2ban-client status sshd, faillock --user
14 Password policy and expiry Done pwquality.conf, login.defs, chage -l
15 A single UID 0; no accounts without a password Done The three audit one-liners
16 Least privilege in sudo Done /etc/sudoers.d/tramontana with Cmnd_Alias
17 MAC: AppArmor profile in enforce Done aa-status, the tramontana profile
18 Service with systemd hardening Done systemd-analyze security: 1.6 OK
19 Secrets out of the configuration files Done LoadCredentialEncrypted, pass
20 TLS: material issued, verified and with renewal tested Done certbot renew --dry-run, check_certificate.sh
21 Persistent logging with rotation Done /var/log/journal, logrotate.d/tramontana
22 File integrity watched, database off the machine Done tramontana-integrity.timer, the checksum on the laptop
23 Auditing of access to configuration and identities Done auditctl -l, keys tramontana_conf and privileges
24 3-2-1 backups, encrypted, with a tested restore Done restic snapshots, check_backup.sh, the runbook
25 Encryption in transit in force Outstanding (08-01) Certificate ready; the reverse proxy is missing. 8080 is not exposed
26 Off-site backup in append-only mode Outstanding Requires a provider decision; it mitigates ransomware
27 Unexplained authorized_keys2 Open The incident response procedure from 06-04 is under way
28 Centralised logs off the server Accepted Current cost; the first investment when there is budget
29 NIDS Accepted It adds little with a single server (reasoned in 06-04)
30 Antivirus Accepted A server with no third-party files; it consumes limited memory
31 A second factor on SSH Accepted Access is already key-based; to be reviewed if the team grows
32 Application security (injection, logic) Out of scope Development's responsibility
33 Physical and hypervisor security Out of scope The provider's responsibility
34 A formal security audit Out of scope Requires an independent third party

Rows 25 to 27 are the genuinely important ones, because they are the ones an honest report does not hide. And rows 32 to 34 delimit what an administrator cannot resolve, which is information the management needs to have.

Maintaining the posture

A hardened system degrades on its own: every change, every new package, every "temporary" rule moves it further from the state you documented. Four practices sustain it:

  1. Periodic review with comparable metrics. Lynis monthly, with its hardening index as a time series alongside the performance baseline from 05-07:
    $ sudo lynis audit system --quiet | grep 'Hardening index'
      Hardening index : 82 [################    ]
    
    From the 68 in 06-04 to the current 82. What matters is not the number: it is that if it drops, there is an explanation to go and find.
  2. Change management. Every configuration modification with its prior copy, its diff -u, its verification and its note in the record. It is the course's convention, and it is the control against the most likely actor in the threat model.
  3. Living documentation in the runbook, off the server: the threat model, this checklist with dates, the inventory of secrets with their last rotation, and the incident response procedure.
  4. A review of the threat model itself when reality changes: one more server, a new service, a new kind of data that starts being processed. A threat model from two years ago describes a system that no longer exists.

Compliance warning

srv-tramontana processes guests' personal data. The GDPR requires technical and organisational measures appropriate to the risk (art. 32), and several of this lesson's decisions — log retention, the scope of auditing over working people's activity, the timescale for deploying encryption in transit in row 25 — have legal implications that are not an administrator's to decide. In a real environment:

  • The security officer must review the threat model and the checklist, and validate the accepted controls.
  • The data protection officer must validate the processing, the retention periods and the residual risk assessment.
  • And this checklist does not replace a formal audit. It is an honest self-assessment, made by whoever administers the system, and by definition it has the blind spot of the person who built it.

Common Mistakes and Tips

  • Hardening with no threat model. It is the cause of half the fragile systems out there: copied measures that break things and protect against nothing that was likely. Half a page of model changes every decision that follows.
  • Touching PAM without a root session open. A syntax error in common-auth can prevent every login, the console included. A root session in another terminal, a prior copy, and a test in a third.
  • Forgetting pam_faillock.so authsucc. Without that line the failure counter is never reset, and the lockout arrives days later with no apparent cause.
  • Forgetting enforce_for_root = 1 in pwquality.conf: the most important account ends up exempt from the password policy.
  • Putting an AppArmor profile straight into enforce. It breaks the service and forces you to discover the missing accesses under pressure. Always complain first, exercise it for real, then enforce.
  • Applying noexec to /tmp without checking. Some installers extract and run there. Test with mount -o remount and exercise apt and your scripts before touching fstab.
  • Editing fstab without mount -a. A broken fstab stops the server booting. It is the safety net from 05-04 and it is still compulsory.
  • Confusing "the package is updated" with "the vulnerability is fixed". While the process has the old library in memory, it is still exposed. needrestart -r l after every update, and lsof | grep DEL as confirmation.
  • Hardening and not verifying. systemd-analyze security improves the score even if you have left the service unable to start. Apply, verify with health_check.sh, and then measure.
  • MemoryDenyWriteExecute on an application with a JIT. Java, Node and .NET will not start. Read what each directive does before copying it.
  • A checklist with no evidence column. "Firewall configured" says nothing; ufw status verbose does. Without evidence, the checklist is a statement of intent.
  • A tip on method. Turn this lesson's audits into audit_hardening.sh using lib/common.sh, with a monthly timer and the principle of silence if all is well. What depends on your memory is not a control.

Exercises

Exercise 1

After applying the hardening drop-in, tramontana.service stops starting:

$ sudo systemctl status tramontana.service --no-pager | head -6
● tramontana.service - Tramontana Bookings
     Active: failed (Result: exit-code) since Tue 2026-08-18 16:04:11 CEST
    Process: 9214 ExecStart=/opt/tramontana/app/tramontana ... (code=killed, signal=SYS)

Describe the diagnostic procedure, identify the most likely cause from the information in the message, and explain how you would narrow down which specific directive causes it without removing them all at once.

Exercise 2

Write the AppArmor profile for the /home/operator/bin/deploy-safe wrapper, which is run via sudo and calls deploy.sh. Reason about what it must permit and — more importantly — what it must deny, and explain which specific attack that profile contains that neither DAC nor the sudoers rule contains.

Exercise 3

Marta asks you for a one-page report for the management: "are we secure?". Write it, drawing on the threat model and the checklist, without unnecessary jargon, honestly saying what is protected, what is not and which decisions need budget or authorisation.

Solutions

Solution 1

The clue is in the message itself: code=killed, signal=SYS. SIGSYS is the signal the kernel sends when a seccomp filter rejects a system call. That points directly at SystemCallFilter, and not at the other directives (which would typically produce EACCES, EPERM or a path failure).

The procedure, from the most informative to the most expensive:

# 1. Confirm the hypothesis and see WHICH call was rejected
$ sudo journalctl -u tramontana.service -n 30 --no-pager | grep -iE 'seccomp|SYS|syscall'
audit: type=1326 audit(1755530651.882:88): auid=4294967295 uid=997 pid=9214
  comm="tramontana" exe="/opt/tramontana/releases/3.2.1/tramontana"
  syscall=318 compat=0 ip=0x7f2a1c4b3e2a code=0x80000000

# 2. Translate the call number into its name
$ ausyscall 318
getrandom

getrandom is the call for obtaining randomness from the kernel — the application uses it for TLS and for session identifiers. It is in @system-service, but I excluded it by adding SystemCallFilter=~@privileged @resources @obsolete, and getrandom belongs to the @resources set in some versions of systemd. The second line, which looked like an improvement, is the one that breaks the service.

How to narrow it down without removing everything, which is the method part of the exercise. The key is that drop-ins can be stacked and overridden separately:

# a) A temporary drop-in that only neutralises the suspect directive.
#    The empty line RESETS the accumulated list; then only the minimum is set.
$ sudo mkdir -p /etc/systemd/system/tramontana.service.d
$ sudo tee /etc/systemd/system/tramontana.service.d/99-diagnostic.conf >/dev/null <<'EOF'
[Service]
SystemCallFilter=
SystemCallFilter=@system-service
EOF
$ sudo systemctl daemon-reload && sudo systemctl restart tramontana.service
$ ~/scripts/health_check.sh; echo "status: $?"
status: 0

It starts. That confirms the problem was the exclusion, not the base set. Now you refine instead of giving up on the filter:

# b) Recover the useful restriction, giving back only what is necessary
$ sudo tee /etc/systemd/system/tramontana.service.d/99-diagnostic.conf >/dev/null <<'EOF'
[Service]
SystemCallFilter=
SystemCallFilter=@system-service
SystemCallFilter=~@privileged @obsolete
SystemCallFilter=getrandom
EOF
$ sudo systemctl daemon-reload && sudo systemctl restart tramontana.service
$ ~/scripts/health_check.sh && systemd-analyze security tramontana.service | tail -1
→ Overall exposure level for tramontana.service: 1.7 OK 🙂

~@privileged @obsolete is kept — which is where nearly all the defensive value lies — @resources is abandoned, and getrandom is added explicitly. Exposure 1.7 instead of 1.6, with the service running: worse hardening on paper and infinitely better in practice, because the 1.6 one did not start.

A general method applicable to any hardening failure:

  1. Read the signal or the error code: SIGSYS → seccomp; EPERM on a file → ProtectSystem/ReadWritePaths; EACCES with apparmor="DENIED" → AppArmor; a failure to map memory → MemoryDenyWriteExecute.
  2. Find the concrete evidence in the journal (syscall=, denied_mask=, the path).
  3. Neutralise one directive with a drop-in with a higher numeric prefix, never by editing the original.
  4. Verify that it starts, refine towards the necessary minimum, and measure again.
  5. Consolidate the drop-in with a definitive name and write down the reason for the exception — just like the commented-out unprivileged_userns_clone in the sysctl file.

Solution 2

# /etc/apparmor.d/home.operator.bin.deploy-safe
abi <abi/4.0>,
include <tunables/global>

/home/operator/bin/deploy-safe {
  include <abstractions/base>
  include <abstractions/bash>

  # The wrapper and the script it invokes
  /home/operator/bin/deploy-safe          r,
  /home/operator/scripts/deploy.sh        rix,
  /home/operator/scripts/lib/common.sh    r,
  /usr/bin/bash                           rix,

  # The specific utilities the script needs, ENUMERATED
  /usr/bin/{tar,curl,ln,sha256sum,systemctl,flock,mktemp,rm,mv,date,logger} rix,

  # Source of the release package: read only
  /srv/tramontana/backups/outgoing/*.tar.gz  r,
  /srv/tramontana/backups/outgoing/*.sha256  r,

  # Deployment destination: writing confined to releases and the link
  /opt/tramontana/releases/                  rw,
  /opt/tramontana/releases/**                rw,
  /opt/tramontana/app                        rw,
  /opt/tramontana/HISTORY                    rw,

  # Log and lock
  /var/log/tramontana/deploy.log              rw,
  /var/lock/tramontana-deploy.lock            rwk,
  /tmp/                                       r,
  /tmp/**                                     rw,

  # Query the service's state after the deployment
  /run/systemd/private                        rw,

  # Explicitly DENIED, so that it is on record and gets logged:
  deny /etc/shadow                            rwklx,
  deny /etc/tramontana/secrets/**             rwklx,
  deny /srv/tramontana/backups/restic/**      rwklx,
  deny /home/operator/.ssh/**                 rwklx,
  deny /home/operator/.password-store/**      rwklx,
  deny /root/**                               rwklx,
  deny /usr/bin/{nc,ncat,socat,ssh,scp,python3,perl} x,
}

What it permits: reading the release package, writing under releases/, changing the app link, writing its log and its lock, and talking to systemd to restart the service. Nothing else.

What it denies, and which attack each denial contains:

Denial The attack it contains
/etc/tramontana/secrets/** The process runs as root via sudo, so DAC would let it read the encrypted credential. AppArmor does not
/srv/tramontana/backups/restic/** A deployment has no reason whatsoever to touch the backups. It contains the deletion of backups by ransomware
/home/operator/.password-store/** Stops the wrapper reaching the source of truth for the secrets
/home/operator/.ssh/** Blocks adding an authorised key — the persistence mechanism AIDE detected in 06-04
x on nc, socat, python3, perl Blocks the reverse shell, which is the first step after obtaining privileged execution

And now the substance of the exercise: what this profile contains that DAC and sudoers do not.

The sudoers rule lets operator run deploy-safe as root. From that moment on, DAC offers no protection at all: root can read /etc/shadow, can read the database credential, can delete the backups, can add itself an SSH key and can launch nc to open a shell outwards. The wrapper was written precisely to avoid granting full root, but the containment depends entirely on the content of the script being correct and staying correct.

That is where the gap is. If deploy.sh has a flaw — an unquoted variable that allows a command to be injected, a tar that extracts a file with ../ in its path — the attacker executes code as root through an authorised route. Neither sudo nor the Unix permissions stop it: from their point of view the execution is legitimate.

The AppArmor profile is the layer that does stop it, because the control does not depend on the user but on the program: whatever code ends up running inside this profile, it will not be able to read the secrets, will not be able to touch the backups, will not be able to write to authorized_keys and will not be able to launch an interpreter to escape. The damage stays confined to the deployment, which is exactly what was intended when the wrapper was written and what until now was only an intention.

That is defence in depth in its most concrete form: sudoers limits who and which command; AppArmor limits what that command can do, even when the command misbehaves.

Solution 3

Security report — srv-tramontana / Tramontana Bookings To: Management · From: Systems Operations · 18 August 2026

In one sentence. The server is reasonably protected against the threats that genuinely affect it, with three outstanding matters that I set out at the end, one of which requires a decision from management.

What we are protecting, in order of importance. First, our guests' personal data: names, dates of stay and amounts. A leak would be irreversible, would harm third parties and has legal consequences. Second, the availability of the booking service, whose cost is economic and bounded in time.

Who we are protecting ourselves from, and what we have done.

  • Automated attacks from the Internet, which are constant. The server accepts connections through two doors only: the administration one — which no longer accepts passwords, only cryptographic keys — and the website's. Everything else is closed by default, and anybody who persists in trying passwords is blocked automatically. In the last month 47 attempts from a single source have been blocked.
  • Theft or leakage of a password. The system's credentials are no longer written in any readable file: they are encrypted and only the program that needs them can use them. We also have a written procedure for changing them without interrupting the service.
  • Our own mistakes, which statistically are the most likely risk. Every modification is made with a prior copy, is checked before being applied and is recorded. An independent system checks every day that no critical file has changed without explanation.
  • Misuse of legitimate access. Each person has only the permissions their job requires, and accesses to the configuration are recorded.

What we do if something fails. We have encrypted backups, with the restore tested — not merely configured — capable of recovering the service in 8 hours with a maximum loss of 4 hours of bookings, timescales agreed with Operations. And a written incident response procedure, kept off the server.

The three outstanding matters.

  1. Web traffic is not yet encrypted. The certificate has been issued and verified, but the component that will present it still has to be deployed, and that is scheduled for the next phase. In the meantime the service is not reachable from the Internet, so the exposure is limited to our internal network. No decision required: it is planned.
  2. An unexplained finding. Our monitoring detected an access configuration file that nobody created. It is under investigation following the established procedure. I will report the outcome; if access to personal data were confirmed, there is a legal obligation to notify it within 72 hours.
  3. An off-site backup protected against deletion. Today, anybody with control of the server could destroy the backups. There is a form of storage that does not allow what has already been stored to be deleted, and it is the best available defence against a data-hostage attack. Requires a management decision: it means contracting external storage.

What is not in our hands. Three things fall outside what systems administration can resolve, and it is worth putting them on record: the internal security of the application itself, which is development's responsibility; physical and infrastructure-provider security; and an attacker with resources far greater than those of a company of our size.

And a necessary caveat. This report is a self-assessment made by whoever administers the system, and it therefore has the blind spot of the person who built it. It does not replace an independent audit. I also recommend that the processing of guests' data and the retention periods for logs be reviewed by the data protection officer.

In summary: the work is done, measured and documented. The only decision I need from management is the one in point 3.

Conclusion

srv-tramontana now has a security posture, and that is qualitatively different from having security measures. There is a written threat model that says what we protect, from whom and what we assume to be out of scope, and every decision in this lesson is justified by it. The attack surface has been reduced: the application listens only locally, and what is not used is masked and purged. Authentication goes through PAM with a password policy, per-account lockout and resource limits — and you tracked down the Match User luis that had been contradicting your own configuration for weeks. AppArmor contains the application in enforce, with a profile that prevents launching a shell or reading the backups even if the process is compromised. The hardening sysctl settings are applied and commented, including the line you left disabled with the reason written beside it. /tmp has no execute permission, checked before it was made persistent. And systemd-analyze security has gone from 3.4 to 1.6 with the service verified and working, which is the only improvement that counts.

The three incidents are closed: the 47 attempts from 203.0.113.44 with fail2ban, the db_password in the clear with systemd-creds and pass, and today the vulnerable libssl that was still loaded in memory — because an updated package is not a fixed vulnerability until the processes know about it. What remains is a 34-row checklist with its evidence column, three open matters stated without decoration and a report for the management that does not promise what it cannot deliver. The Lynis index has risen from 68 to 82, and the important thing about that number is not its value but that it is now a time series somebody watches.

And with that, Module 6 closes: you configured the network persistently with netplan try's automatic rollback; you hardened SSH with ed25519 keys, no root and no passwords, and learned to read sshd -T -C to find out what configuration really applies to somebody; you raised an allowlist firewall over nftables and learned the order it is applied in so as not to lock yourself out; you set up intrusion detection with AIDE and auditd, with the database protected off the server, and an incident response procedure with its uncomfortable rule; you got the secrets out of the configuration files and prepared the TLS cryptographic material with renewal tested and expiry monitored; and today you have brought all of that together into a coherent, documented posture.

So far you have operated one server, and you have operated it by hand. You know how to use it, administer it and defend it, but there are two things you still cannot do: look inside and multiply it. In Module 7: Advanced Topics you do both. You will see the complete boot process — from UEFI to GRUB, to the initramfs and to systemd — and learn to recover a system that will not boot, which is the skill that separates the people who reinstall from the people who fix. You will learn advanced diagnosis with strace, perf and eBPF to answer "what exactly is this process doing?" when the counters are not enough (and you will discover that the ptrace_scope you have just set is part of the problem statement). You will tune the kernel with sysctl, this time for performance. And then the multiplication begins: virtualisation with KVM and libvirt, containers with Docker and why they are isolated processes and not small machines, automation with Ansible — which will turn all the manual work of modules 5 and 6 into versioned, reproducible code, and the 8-hour RTO into considerably less — and finally high availability and load balancing, where srv-tramontana stops being a single point of failure. Update your VM snapshot, keep the runbook off the machine, and I will see you in Module 7.

Linux Course: From Beginner to System Administrator

Module 1: Introduction to Linux

Module 2: Basic Linux Commands

Module 3: Advanced Command-Line Skills

Module 4: Shell Scripting

Module 5: System Administration

Module 6: Networking and Security

Module 7: Advanced Topics

Module 8: Practical Projects

© Copyright 2026. All rights reserved