The previous five lessons have added security measures one at a time: a controlled network, SSH with keys, a firewall, intrusion detection, encrypted secrets and TLS. Each one solved a specific problem. This lesson does something different: it turns them into a security posture, which is a coherent set of decisions justified by an explicit threat model, with a written record of what is protected, what has been consciously accepted and what falls outside an administrator's remit.
The difference is not rhetorical. With no threat model, hardening degenerates into cargo cult: recipes found in guides get applied because they "sound secure", things break without anybody understanding why, and you end up with a fragile system and a false sense of control. With a threat model, every measure has a reason and every omission is a conscious decision.
You also close the third outstanding incident here — the services still using the vulnerable libssl loaded in memory — and settle the debt you left in 05-01: PAM, the subsystem that really decides how somebody authenticates on this system.
Warning up front. Two of the sections in this lesson — PAM and secure mounts — can leave you unable to log in to your own server if you get them wrong. In both cases the safe procedure is given. Do not skip it: the lab exists precisely so that you learn this where mistakes cost nothing.
Contents
- The five principles that order everything so far
- Tramontana's threat model
- Reducing the attack surface
- Hardening accounts and the password policy
- PAM: how this system really authenticates
- Mandatory access control with AppArmor
- Kernel hardening with sysctl
- Secure mounts
- Vulnerability management and closing the openssl incident
- systemd hardening revisited
- Backups as the last control against ransomware
- A hardening checklist and maintaining the posture
The five principles that order everything so far
Everything you have done in this module answers, without our having said so, to five principles. Naming them lets you apply them to new situations instead of repeating recipes:
| Principle | What it demands | Where you have already applied it |
|---|---|---|
| Least privilege | Every process and every person with exactly the permissions they need, not one more | svc-tramontana with no shell, sudoers with specific commands, an empty CapabilityBoundingSet= |
| Minimal attack surface | What does not exist cannot be attacked | A server with no desktop, port 8080 closed to the outside, an allowlist policy in ufw |
| Defence in depth | Several independent layers; one of them failing must not be enough | Firewall + hardened SSH + AIDE + auditd + encrypted secrets |
| Fail safe | When something breaks, it must end up closed, not open | policy drop by default in nftables, set -euo pipefail in the scripts |
| Secure by default | The initial state must be the restrictive one | umask 027, 600 permissions on the secrets, PasswordAuthentication no |
The hardest to apply in practice is the second, because it requires taking things away, and taking things away is frightening. The most forgotten is the fourth: a lot of "secure" configuration fails open, and that is worse than not having it, because nobody will notice.
Tramontana's threat model
A threat model answers three questions. Half a page is enough, and it is the half page that makes everything else make sense.
What are we protecting? Two things, in this order:
- The guests' personal data (names, dates of stay, amounts in
bookings.csvand in the database). Its compromise is harm to third parties, has legal consequences under the GDPR and is irreversible: once leaked, it cannot be "recovered". - The availability of the booking service. Its interruption has a direct economic cost, bounded in time.
From whom? Four realistic actors, ordered by likelihood:
| Threat | Likelihood | Capability | The controls we set against it |
|---|---|---|---|
| Automated scanning from the Internet | Constant | Low: it exploits the known and unpatched | Firewall, fail2ban, security updates, SSH without passwords |
| A leaked or reused credential | Medium | High: it gets in as a legitimate user | Keys instead of passwords, encrypted secrets, rotation, auditd |
| Your own configuration mistake | High | Variable, sometimes total | --dry-run, copies before editing, netplan try, mount -a, AIDE |
| Abuse of legitimate access | Low | High within their remit | Least privilege in sudoers, ACLs, access auditing |
Look at the third row. The most likely actor is you, and that observation explains why half a dozen of the course's conventions — copy before editing, verify with diff -u, run a dry run, netplan try with automatic rollback, mount -a before rebooting — are security controls in their own right, and not stylistic fussiness.
What do we assume is out of scope? Saying it is as important as the above, because it delimits what is not being protected:
- A state-resourced attacker or a compromised supply chain. That is beyond any measure an administrator can take on a small company's server.
- Physical access to the machine. LUKS encryption mitigates disk theft, but an attacker with prolonged physical access and the machine powered on owns the system.
- A vulnerability in the application itself (SQL injection, business logic). That is development's responsibility, not administration's, and system hardening only limits the subsequent damage.
- A compromise of the cloud provider or the hypervisor.
And now the underlying criterion: with this model, investing in a NIDS adds little (you already reasoned that in 06-04) and investing in change detection and authentication control adds a great deal. That is making decisions, not following a list.
Reducing the attack surface
What does not exist cannot be attacked, needs no patches and appears in no CVE. It is the principle with the best effort-to-result ratio.
What is listening
$ sudo ss -tulpn
Netid State Local Address:Port Peer Address:Port Process
udp UNCONN 127.0.0.54:53 0.0.0.0:* users:(("systemd-resolve",pid=712,fd=17))
udp UNCONN 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=712,fd=13))
tcp LISTEN 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=712,fd=14))
tcp LISTEN 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=894,fd=3))
tcp LISTEN 0.0.0.0:8080 0.0.0.0:* users:(("tramontana",pid=4102,fd=6))
tcp LISTEN 10.0.2.15:5432 0.0.0.0:* users:(("postgres",pid=1105,fd=5))Read each line asking "does it have to be there, and on that address?":
systemd-resolveon127.0.0.53and127.0.0.54: local, correct.sshdon0.0.0.0:22: necessary, and already hardened.tramontanaon0.0.0.0:8080: listening on every interface. The firewall blocks it from outside, but that is defence in a single layer. The right thing is for the application to listen only on127.0.0.1, because when the reverse proxy from 08-01 arrives it will be the one talking to it, and that way the firewall becomes the second line instead of the only one.postgreson10.0.2.15:5432: already restricted to the internal interface. Correct.
The first fix, then:
$ sudo cp -p /etc/tramontana/app.conf /etc/tramontana/app.conf.bak-$(date +%F)
$ sudo chattr -i /etc/tramontana/app.conf
$ echo 'listen=127.0.0.1' | sudo tee -a /etc/tramontana/app.conf
$ sudo chattr +i /etc/tramontana/app.conf
$ sudo systemctl restart tramontana.service
$ sudo ss -tulpn | grep 8080
tcp LISTEN 127.0.0.1:8080 0.0.0.0:* users:(("tramontana",pid=4318,fd=6))That is defence in depth applied: it now takes two failures — a badly written firewall rule and a listener that is too broad — to expose 8080. Before, one was enough.
What is enabled
$ systemctl list-unit-files --state=enabled --type=service --no-pager
UNIT FILE STATE PRESET
apparmor.service enabled enabled
auditd.service enabled enabled
cron.service enabled enabled
fail2ban.service enabled enabled
ModemManager.service enabled enabled
multipathd.service enabled enabled
postgresql.service enabled enabled
ssh.service enabled enabled
tramontana.service enabled enabled
unattended-upgrades.service enabled enabledModemManager manages modems and mobile broadband. On a server virtual machine it makes no sense at all. And here it is worth understanding the difference between the two ways of switching something off, because they are not equivalent:
| Operation | What it does | Can it come back on its own |
|---|---|---|
disable |
Deletes the [Install] links; it does not start at boot |
Yes: if another unit declares it in Wants= |
mask |
Links the unit to /dev/null; it cannot be started |
No: neither by hand nor through a dependency |
$ sudo systemctl disable --now ModemManager.service
$ sudo systemctl mask ModemManager.service
Created symlink /etc/systemd/system/ModemManager.service → /dev/null.mask for what must never start; disable for what you might want to start by hand one day. And the next step, when you are sure, is to uninstall the package: a binary that is not on the disk has no vulnerabilities.
On the desktop, to close the section: a graphical environment adds hundreds of packages, an X or Wayland server, a session manager and a browser — each with its own surface and its own patching calendar. srv-tramontana not having one is not austerity, it is the biggest surface reduction you can make on a server, and that is why the Ubuntu Server installer does not offer it by default.
Hardening accounts and the password policy
Three audits worth keeping in a script, because they catch serious mistakes and are one line each:
# 1. Accounts with UID 0: there must be only one, root
$ awk -F: '($3 == 0) {print $1}' /etc/passwd
root
# 2. Accounts with no password (empty second field in shadow): serious
$ sudo awk -F: '($2 == "") {print $1 " HAS NO PASSWORD"}' /etc/shadow
# 3. System accounts (UID < 1000) with a valid login shell
$ awk -F: '($3 < 1000 && $7 !~ /(nologin|false|sync)$/) {print $1, $3, $7}' /etc/passwd
root 0 /bin/bash
sync 4 /bin/syncAll three results are correct: a single UID 0, no account without a password, and the only two system accounts with a shell are root — needed for recovery — and sync, which is a harmless historical case. The fact that svc-tramontana does not appear in the third list confirms that the -s /usr/sbin/nologin from 05-01 did its job.
The default policy lives in /etc/login.defs, and it affects the accounts created from now on:
# /etc/login.defs (modified values)
PASS_MAX_DAYS 365 # maximum age
PASS_MIN_DAYS 1 # stops it being changed several times in a row to get back to the old one
PASS_WARN_AGE 14 # days of warning before it expires
UMASK 027 # consistent with the course's convention
SHA_CRYPT_MIN_ROUNDS 5000
ENCRYPT_METHOD YESCRYPTPASS_MIN_DAYS 1 is less obvious than the rest: without it, somebody who is required to change their password can change it five times in a row to exhaust the history and go back to the original. It is an old trick and it still works wherever this is not set.
And for the accounts that already exist, chage:
$ sudo chage -M 365 -m 1 -W 14 luis
$ sudo chage -l luis
Last password change : Aug 04, 2026
Password expires : Aug 04, 2027
Password inactive : never
Account expires : never
Minimum number of days between password change : 1
Maximum number of days between password change : 365
Number of days of warning before password expires : 14
# The intern's account has an end-of-placement date: it expires automatically
$ sudo chage -E 2026-12-31 intern
$ sudo chage -l intern | grep 'Account expires'
Account expires : Dec 31, 2026chage -E deserves attention: it is the correct way to manage a temporary account. It expires on its own on the expected date, without depending on somebody remembering to close it — and forgotten leavers are one of the most common routes to compromise there is.
PAM: how this system really authenticates
In 05-01 we deferred PAM. Now is the moment, because it is the piece that really decides who gets into this system and on what conditions.
PAM (Pluggable Authentication Modules) is a layer of indirection: the programs that need to authenticate — login, sshd, sudo, su, passwd — do not implement the logic themselves, they ask PAM. That makes it possible to change the policy (demand strong passwords, lock out after failures, add a second factor) without modifying or recompiling any program. It is the reason it exists.
The structure
Each service has its file in /etc/pam.d/, and there are common files that the rest include:
$ ls /etc/pam.d/ | head -12
common-account
common-auth
common-password
common-session
cron
login
passwd
sshd
su
sudoEach line has three parts: type, control and module.
The four types, which are four different questions answered at different moments:
| Type | The question it answers |
|---|---|
auth |
Are they who they say they are? (password, key, second factor) |
account |
Are they allowed in right now? (expired account, time of day, origin) |
password |
Is the new password they want to set acceptable? |
session |
What has to be set up and cleaned up around the session (limits, logging, home) |
Confusing auth with account is the most common misunderstanding: a correct password on an expired account passes auth and fails account.
The controls determine what happens according to the module's result:
| Control | If the module fails | If it succeeds |
|---|---|---|
required |
The stack fails, but the rest still run | Continues |
requisite |
It fails and stops right there | Continues |
sufficient |
Ignored | Immediate success, if no earlier required failed |
optional |
Ignored (unless it is the only one) | Continues |
required versus requisite has a rationale that is not obvious: required goes on running the stack so as not to reveal at which step it failed. If an attacker could tell "no such user" from "wrong password" by the response time or the message, they would have an oracle for enumerating users.
The modern, more precise syntax uses square brackets: [success=ok default=die] lets you specify the action for each return code. It is what you will see in Ubuntu's files.
The procedure for not locking yourself out
Before you touch a single line. A syntax error in common-auth can prevent every login, the local console included, and then the only way out is recovery mode:
- Leave a root session open in another terminal, and do not close it until you have finished:
$ sudo -i # (do not close this session) - Copy the file before editing it, following the course's convention:
$ sudo cp -p /etc/pam.d/common-password /etc/pam.d/common-password.bak-$(date +%F) - Test in a third terminal, never in the one holding the root session.
- Keep the VM console to hand and know how to enter recovery mode (covered in depth in 07-01).
It is exactly the same discipline as netplan try and the second SSH session: never close the door you are coming in through.
pam_pwquality: demanding decent passwords
# /etc/security/pwquality.conf
minlen = 12 # minimum length
minclass = 3 # at least 3 of: lower case, upper case, digits, symbols
maxrepeat = 3 # no more than 3 identical consecutive characters
dictcheck = 1 # rejects dictionary words
usercheck = 1 # rejects passwords containing the username
enforce_for_root = 1 # root too: without this, root is exempt
retry = 3$ passwd luis
New password: tramontana2026
Bad password: it is based on a dictionary word.
New password: Kj8-mQ2vX9pL
passwd: password updated successfullyTwo underlying comments. The first: minlen = 12 with minclass = 3 is more sensible than the "8 characters with an upper-case letter, a number and a symbol" policies that produce Passw0rd! — length contributes far more than forced complexity, and that is what current guidance says (NIST SP 800-63B). The second: enforce_for_root = 1 is easy to forget, and without it the most important account on the system is the only one exempt from the policy.
pam_faillock: locking out after failed attempts
It complements fail2ban, which acts on the IP; faillock acts on the account, so it also covers attempts from the console or from different IPs.
# /etc/security/faillock.conf
deny = 5 # locks out after 5 failures
fail_interval = 900 # occurring within 15 minutes
unlock_time = 600 # automatic unlock after 10 minutes
even_deny_root # root too
root_unlock_time = 60 # but root unlocks in 1 minute: avoids a total lockout
audit # records the attempt
silent # does not reveal to the attacker that the account is locked# /etc/pam.d/common-auth
auth required pam_faillock.so preauth
auth [success=1 default=ignore] pam_unix.so nullok
auth [default=die] pam_faillock.so authfail
auth sufficient pam_faillock.so authsucc
auth requisite pam_deny.so
auth required pam_permit.soThe order matters and it is not intuitive: preauth checks whether the account is already locked before asking for credentials, authfail counts the failure, and authsucc clears the counter after a successful login. Missing out authsucc produces a slow but inevitable lockout, because the counter is never reset.
And the combination of even_deny_root with root_unlock_time = 60 is deliberate: protecting root without creating the possibility of a total system lockout.
$ faillock --user luis
luis:
When Type Source Valid
2026-08-18 14:22:11 RHOST 203.0.113.44 V
2026-08-18 14:22:14 RHOST 203.0.113.44 V
$ sudo faillock --user luis --reset # unlock by handpam_limits: resource limits
# /etc/security/limits.conf
# domain type item value
* hard nproc 4096 # curbs a process bomb
* hard core 0 # do not generate dumps: they can contain secrets
svc-tramontana soft nofile 8192
svc-tramontana hard nofile 16384Two of these lines are security measures rather than performance ones. nproc limits the damage of a loop that creates processes without control — accidental or otherwise. And core 0 avoids memory dumps, which contain everything the process had in RAM at that moment: including the password you went to such lengths to encrypt in 06-05.
Careful with the scope: pam_limits applies to sessions that go through PAM. For a systemd service, the limits go in the unit (LimitNOFILE, LimitNPROC), as you saw in 05-07.
A second factor, briefly
libpam-google-authenticator adds a TOTP code to SSH authentication. It is effective against credential theft, and on a server with key-based access the marginal benefit is smaller than on one with passwords. If it is adopted, there are two things you cannot forget: keeping the recovery codes off the server, and deciding what happens if the TOTP device is lost. Without that, it is an elegant way of locking yourself out.
Closing the finding: the PasswordAuthentication exception
In 06-04 the logs showed that luis logged in with password, even though you disabled it in 06-02. Time to track it down, starting with the authoritative source:
$ sudo sshd -T | grep -i -E 'passwordauth|kbdinteractive'
passwordauthentication no
kbdinteractiveauthentication noThe global configuration is correct. But sshd -T with no arguments shows the global configuration; Match blocks are conditional and do not appear there. You have to ask about the specific case:
$ sudo grep -rn -A3 'Match' /etc/ssh/sshd_config /etc/ssh/sshd_config.d/
/etc/ssh/sshd_config.d/60-tramontana.conf:12:Match User luis
/etc/ssh/sshd_config.d/60-tramontana.conf-13- PasswordAuthentication yes
$ sudo sshd -T -C user=luis,host=laptop-luis,addr=10.0.2.44 | grep -i passwordauth
passwordauthentication yesThere it is. A Match User luis block with the exception, put in place "temporarily" while Luis generated his key, and never withdrawn. It is the most likely actor in the threat model — your own mistake — in its most characteristic form.
$ sudo cp -p /etc/ssh/sshd_config.d/60-tramontana.conf{,.bak-$(date +%F)}
$ sudo sed -i '/^Match User luis$/,+1d' /etc/ssh/sshd_config.d/60-tramontana.conf
$ sudo sshd -t && echo "syntax OK"
syntax OK
$ sudo sshd -T -C user=luis,host=laptop-luis,addr=10.0.2.44 | grep -i passwordauth
passwordauthentication no
$ sudo systemctl reload sshNote sshd -T -C: the -C option evaluates the configuration for a specific context of user, host and address, resolving the Match blocks. It is the only reliable way to know what configuration applies to somebody, and it deserves a place in your memory: half of the baffling SSH problems are a forgotten Match.
Before reloading, the usual discipline: sshd -t validates, the second session is open, and reload is used instead of restart.
Mandatory access control with AppArmor
The permissions you have known since 02-07 are DAC (Discretionary Access Control): discretionary because the owner of a resource decides who gets access. They have a structural limit: a process can do everything its user can do. If Tramontana's application is compromised, the attacker can read anything readable by svc-tramontana and write anywhere that user can write.
MAC (Mandatory Access Control) adds a layer the owner cannot relax: a policy, defined by the administrator and enforced by the kernel, that says what this program can do, regardless of who runs it.
| DAC (Unix permissions) | MAC (AppArmor / SELinux) | |
|---|---|---|
| Who decides | The file's owner | The administrator, in the policy |
| Unit of control | User and group | Program (profile) |
| Can the process relax it | Yes, within its user's remit | No |
| What it contains | A compromise of the process | A compromise of the process and of its user |
$ sudo aa-status
apparmor module is loaded.
32 profiles are loaded.
28 profiles are in enforce mode.
/usr/bin/man
/usr/sbin/sshd
...
2 profiles are in complain mode.
2 processes have profiles defined.The two modes are the key to the working method:
complain: it does not block, it only records what it would have blocked. It is the learning mode.enforce: it blocks. It is the production mode.
And the professional procedure is always the same: write the profile, put it in complain, exercise the application for real, collect the denials and extend the profile, and only then switch to enforce. Starting in enforce means breaking the service and discovering the missing accesses in the worst possible way.
A profile for Tramontana's application
# /etc/apparmor.d/opt.tramontana.app.tramontana
abi <abi/4.0>,
include <tunables/global>
/opt/tramontana/releases/*/tramontana {
include <abstractions/base>
include <abstractions/nameservice>
include <abstractions/openssl>
# Its own binary: read and execute
/opt/tramontana/releases/*/tramontana mr,
/opt/tramontana/releases/*/** r,
/opt/tramontana/app/** r,
# Configuration: read only. Even if the process is compromised,
# it cannot modify its own configuration.
/etc/tramontana/app.conf r,
# Credential delivered by systemd (the service's private tmpfs)
/run/credentials/tramontana.service/* r,
# Its own logs: create and append, never overwrite or delete
/var/log/tramontana/ r,
/var/log/tramontana/*.log rw,
# Data uploaded by the guests: read and write
/opt/tramontana/shared/uploads/ r,
/opt/tramontana/shared/uploads/** rwk,
# Network: TCP over IPv4/IPv6 only. No raw unix, no raw packets.
network inet stream,
network inet6 stream,
# Nothing else is permitted: AppArmor denies by default.
# In particular, the following are implicitly DENIED:
# /etc/shadow, /home/**, /srv/tramontana/backups/**,
# /root/**, executing /bin/sh and loading modules.
}That last comment is what makes the profile valuable. An attacker running code in the application's context cannot launch a shell, cannot read the backups, cannot touch the home directories and cannot modify its own configuration — things DAC would partly allow them to do, and which are precisely the first steps of any post-exploitation.
# 1. Load in learning mode
$ sudo apparmor_parser -r /etc/apparmor.d/opt.tramontana.app.tramontana
$ sudo aa-complain /opt/tramontana/releases/*/tramontana
# 2. Exercise the application for real: requests, a file upload, a deployment
$ curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/houses
200
$ ~/scripts/health_check.sh; echo "status: $?"
status: 0
# 3. Collect the recorded denials
$ sudo journalctl -k --since "10 min ago" | grep 'apparmor="ALLOWED"'
kernel: audit: type=1400 apparmor="ALLOWED" operation="open"
profile="/opt/tramontana/releases/*/tramontana" name="/opt/tramontana/shared/templates/invoice.html"
pid=4318 comm="tramontana" requested_mask="r" denied_mask="r"In complain mode the marker is ALLOWED with a denied_mask: it means "this would have been blocked". Here a legitimate access the profile did not contemplate has turned up — the templates — so it is added and the cycle repeats:
# 4. Extend it with assistance if you prefer, and switch to enforce
$ sudo aa-logprof
$ sudo aa-enforce /opt/tramontana/releases/*/tramontana
Setting /opt/tramontana/releases/*/tramontana to enforce mode.
$ sudo systemctl restart tramontana.service
$ ~/scripts/health_check.sh; echo "status: $?"
status: 0From here on, a real denial appears as apparmor="DENIED", and it is the first thing to look at when the service starts failing after a deployment that changes paths:
SELinux, for anybody heading to RHEL:
| AppArmor | SELinux | |
|---|---|---|
| Distributions | Ubuntu, Debian, SUSE | RHEL, Fedora, CentOS Stream |
| Identifies objects by | The file's path | A label on the inode |
| Learning curve | Gentle; readable profiles | Steep; more powerful |
| Diagnosis | journalctl + aa-logprof |
ausearch + audit2allow, restorecon |
| Permissive mode | Per profile (complain) |
Global or per domain |
The path-versus-label difference has a practical consequence: in SELinux, moving a file can change its context and break access (hence restorecon); in AppArmor, a symbolic link or a bind mount can bypass a path-based rule. Each model has its weak spot.
Kernel hardening with sysctl
In 06-03 you applied the network sysctl settings. These are the kernel's own, and they are purely about security: the performance ones are covered in 07-03.
# /etc/sysctl.d/60-hardening.conf
# Only root can read the kernel buffer. Stops memory addresses and hardware
# details useful for building an exploit from leaking.
kernel.dmesg_restrict = 1
# Hides kernel pointers in /proc and in the logs.
kernel.kptr_restrict = 2
# Full ASLR: randomises the address space, the heap included.
# 2 is the default in Ubuntu; it is declared here to be explicit.
kernel.randomize_va_space = 2
# Prevents creating hard links to files you do not own and following other
# people's symbolic links in directories with the sticky bit.
# It closes a whole family of race attacks in /tmp.
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
# The same protection for FIFOs and regular files in world-writable
# directories.
fs.protected_fifos = 2
fs.protected_regular = 2
# Only root can use BPF. It greatly reduces that subsystem's surface.
kernel.unprivileged_bpf_disabled = 1
# A process can only debug its direct descendants.
# It stops a compromised process reading the memory of another one belonging to
# the same user (and with it any secrets that other one has loaded).
kernel.yama.ptrace_scope = 1
# Do not let unprivileged users create user namespaces.
# CAREFUL: it breaks unprivileged containers. Commented out because in
# 07-05 you are going to need it; uncomment only on servers with no containers.
#kernel.unprivileged_userns_clone = 0$ sudo sysctl --system 2>&1 | grep -A9 '60-hardening'
* Applying /etc/sysctl.d/60-hardening.conf ...
kernel.dmesg_restrict = 1
kernel.kptr_restrict = 2
kernel.randomize_va_space = 2
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
fs.protected_fifos = 2
fs.protected_regular = 2
kernel.unprivileged_bpf_disabled = 1
kernel.yama.ptrace_scope = 1
$ sysctl kernel.yama.ptrace_scope
kernel.yama.ptrace_scope = 1Two warnings to note down, because they are the two ways this can bite you later:
kernel.yama.ptrace_scope = 1can break debuggers.straceorgdbagainst a process that is already running and is not your child will stop working withoutsudo. In 07-02 you are going to use exactly those tools; remember that this parameter is the explanation if you getOperation not permitted.kernel.unprivileged_userns_clone = 0breaks unprivileged containers, and in 07-05 you are going to set up Docker. That is why it is left commented out with the reason written beside it. A hardening parameter with a note explaining why it is not enabled is quality documentation; deleting it and forgetting the reason is not.
And the general criterion: every line in this file carries its comment. A sysctl.d with fifteen values and no explanation is impossible to maintain, because nobody — you included, in six months — will know whether any of them can be removed.
Secure mounts
Three mount options that limit the damage in the directories where anybody can write:
| Option | Effect |
|---|---|
noexec |
Nothing can be executed from there |
nosuid |
SUID/SGID bits are ignored |
nodev |
Device files are ignored |
The use case is direct: an attacker who manages to write a file usually needs to run it, and /tmp is the place where they can almost always write. With noexec, that step fails.
And this is where the lesson insists on something the hardening guides tend to leave out: noexec on /tmp breaks real things. Several installers extract to /tmp and run from there; so do some language package managers. You have to check first:
# 1. Check the impact live, without touching fstab
$ sudo mount -o remount,noexec,nosuid,nodev /tmp
# 2. Exercise what might break
$ sudo apt install --reinstall -y tree >/dev/null && echo "apt: OK"
apt: OK
$ ~/scripts/deploy.sh --dry-run 3.2.1 && echo "deployment: OK"
deployment: OK
$ sudo needrestart -r l >/dev/null && echo "needrestart: OK"
needrestart: OK
# 3. And check that the restriction really does something
$ printf '#!/bin/sh\necho hello\n' > /tmp/test.sh && chmod +x /tmp/test.sh
$ /tmp/test.sh
bash: /tmp/test.sh: Permission deniedThat last block is the positive verification: it is not enough that nothing has broken, you have to check that the measure does something. With both checks done, it is made persistent:
# /etc/fstab
UUID=3f8a... /tmp ext4 defaults,noatime,noexec,nosuid,nodev 0 2
/tmp /var/tmp none bind,noexec,nosuid,nodev 0 0
tmpfs /dev/shm tmpfs defaults,noexec,nosuid,nodev 0 0$ sudo systemctl daemon-reload
$ sudo mount -a && echo "fstab OK"
fstab OK
$ findmnt -o TARGET,OPTIONS /tmp /var/tmp /dev/shm
TARGET OPTIONS
/tmp rw,noexec,nosuid,nodev,relatime
/var/tmp rw,noexec,nosuid,nodev,relatime
/dev/shm rw,noexec,nosuid,nodevmount -a before rebooting is still compulsory, for the reason you have known since 05-04: a badly written fstab stops the server booting.
Vulnerability management and closing the openssl incident
Reading a CVE without fooling yourself
A CVE is the unique identifier of one specific vulnerability. Its CVSS is a score from 0 to 10 accompanied by a vector describing how it is exploited:
CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H → 9.8 CRITICAL
│ │ │ │ │ │ └─ Confidentiality/Integrity/Availability: high
│ │ │ │ │ └─ Scope: unchanged
│ │ │ │ └─ User interaction: none
│ │ │ └─ Privileges required: none
│ │ └─ Attack complexity: low
│ └─ Attack vector: networkAnd the point to internalise: the base score is not the real risk to you. A 9.8 in a component you do not have installed, or that is not exposed, or that is already mitigated by another layer, is less urgent than a 6.5 in the service facing the Internet. The vector is more informative than the number: AV:N (exploitable over the network) with no privileges and no user interaction is what forces you to act today.
$ apt list --upgradable
$ pro security-status
1543 packages installed:
1489 packages from Ubuntu Main/Restricted repository
54 packages from Ubuntu Universe/Multiverse repository
Ubuntu Pro is not attached. Ubuntu Pro would provide security updates for
54 packages until 2034.
$ sudo apt install debsecan && debsecan --suite noble --format summary | head -5Closing the incident
On 18 August at 06:12, unattended-upgrades updated openssl and libssl3t64. The package is up to date on disk, but the processes that started earlier still have the old library loaded in memory. It is a vulnerability with paperwork: the report says it is patched and the service is still exposed.
$ grep -E 'openssl|libssl' /var/log/apt/history.log | tail -2
Start-Date: 2026-08-18 06:12:03
Upgrade: libssl3t64:amd64 (3.0.13-0ubuntu3.4, 3.0.13-0ubuntu3.5), openssl:amd64 (3.0.13-0ubuntu3.4, 3.0.13-0ubuntu3.5)
# The direct proof: processes with the OLD library (marked DEL: deleted from
# disk but still mapped in memory)
$ sudo lsof 2>/dev/null | grep -E 'DEL.*libssl'
postgres 1105 postgres DEL REG 253,0 /usr/lib/x86_64-linux-gnu/libssl.so.3
sshd 894 root DEL REG 253,0 /usr/lib/x86_64-linux-gnu/libssl.so.3
$ sudo apt install needrestart
$ sudo needrestart -r l
Scanning processes...
Scanning candidates...
Scanning linux images...
Services to be restarted:
systemctl restart [email protected]
systemctl restart ssh.service
Service restarts being deferred:
(none)
No containers need to be restarted.
No user sessions are running outdated binaries.
No VM guests are running outdated hypervisor (qemu) binaries on this host.Two services. The order matters, and for two different reasons:
# 1. The database first, and with the application ready to reconnect.
# Restarting postgresql cuts tramontana.service's open connections.
$ sudo systemctl restart [email protected]
$ sudo systemctl restart tramontana.service
$ ~/scripts/health_check.sh; echo "status: $?"
status: 0
# 2. SSH last, with RELOAD rather than restart, and with the second session open.
# reload does not cut existing sessions; restart should not either, but
# reload is the correct operation and there is no reason to take the risk.
$ sudo sshd -t && sudo systemctl reload sshAn important nuance about ssh: reload re-reads the configuration, but it does not replace the library loaded in the master process. For that you have to restart the daemon — which does not cut sessions already established, because each one lives in its own child process. With the second session open as a safety net:
$ sudo systemctl restart ssh
$ sudo systemctl is-active ssh
active
# Verify from the second session that you can open a third BEFORE closing anythingAnd the closing verification:
$ sudo lsof 2>/dev/null | grep -c -E 'DEL.*libssl'
0
$ sudo needrestart -r l
No services need to be restarted.
$ sudo ss -tulpn | grep -E ':22|:5432|:8080'
tcp LISTEN 0.0.0.0:22 users:(("sshd",pid=8841,fd=3))
tcp LISTEN 10.0.2.15:5432 users:(("postgres",pid=8902,fd=5))
tcp LISTEN 127.0.0.1:8080 users:(("tramontana",pid=8977,fd=6))That 0 is the closure. The second incident is closed, and with it all three.
So that it does not happen again, needrestart is configured in list mode and integrated into the review:
# /etc/needrestart/needrestart.conf
$nrconf{restart} = 'l'; # only list, never restart on its own
$nrconf{kernelhints} = 1; # warn if the running kernel is not the installed one# Add to health_check.sh: restarts pending after an update
$ sudo needrestart -r l -p >/dev/null; echo "needrestart exit code: $?"
needrestart exit code: 0The decision not to restart automatically is deliberate and consistent with the unattended-upgrades of 05-03: an unsupervised restart of the database at six in the morning is a service interruption Marta must know about, not a side effect.
systemd hardening revisited
In 05-05 you hardened tramontana.service. Now it gets measured and improved:
$ systemd-analyze security tramontana.service | tail -12
→ Overall exposure level for tramontana.service: 3.4 MEDIUM 🙂3.4 MEDIUM for a network service is not bad, and there is room. The missing directives, each with its reason:
# /etc/systemd/system/tramontana.service.d/hardening.conf
[Service]
# The service does not need to tune the kernel or load modules.
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
# It must not manipulate the cgroup hierarchy.
ProtectControlGroups=yes
# It must not create SUID/SGID files: it closes a classic persistence route.
RestrictSUIDSGID=yes
# Prevents changing the process's "personality" to emulate another architecture.
LockPersonality=yes
# Only system calls typical of a normal service. The rest are rejected
# with EPERM in the kernel, before reaching the process's code.
SystemCallFilter=@system-service
SystemCallFilter=~@privileged @resources @obsolete
SystemCallArchitectures=native
# Only the socket families it needs.
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictNamespaces=yes
RestrictRealtime=yes
# No access to devices or to /home
PrivateDevices=yes
ProtectHome=yes
ProtectProc=invisible
ProcSubset=pid
# CAREFUL: MemoryDenyWriteExecute breaks JIT runtimes
# (Java, Node.js, .NET, some Python). Only if the application is native.
MemoryDenyWriteExecute=yesSystemCallFilter deserves an explanation because it is the highest-value directive: it installs a seccomp filter in the kernel that rejects system calls outside the permitted set. An exploit that achieves code execution inside the process finds that mount, ptrace, init_module or setuid fail with EPERM — not because of missing user permissions, but because the kernel does not accept them from this process. That is real containment, not cosmetic configuration.
And the warning about MemoryDenyWriteExecute is the kind of detail that separates a useful guide from a copied list: it stops a memory page being writable and executable at the same time, which blocks many exploitation techniques, and it breaks any engine with just-in-time compilation. If Tramontana's application were Java or Node, this line would stop it starting at all.
$ sudo systemctl daemon-reload && sudo systemctl restart tramontana.service
# Verify that the service STILL WORKS: hardening and breaking is not hardening
$ ~/scripts/health_check.sh; echo "status: $?"
status: 0
$ curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/houses
200
$ systemd-analyze security tramontana.service | tail -3
→ Overall exposure level for tramontana.service: 1.6 OK 🙂From 3.4 MEDIUM to 1.6 OK, with the service running and verified. The order — apply, verify, measure — is the same as in credential rotation and in performance diagnosis. Applying without verifying is how you discover at three in the morning that the hardening took the service down.
Backups as the last control against ransomware
When everything above fails, the backups remain. And against ransomware there is a detail that is often overlooked: a permanently mounted backup is reachable by the ransomware, because the process encrypting the system's files also encrypts those on the mounted volume. A backup encrypted by the attacker is not a backup.
The three properties that make a backup resistant:
| Property | How it is achieved | Status at Tramontana |
|---|---|---|
| Offline or immutable | Disconnected media, or storage with enforced retention | Outstanding for the 3-2-1 off-site destination |
| A separate credential | The repository password is not reachable from the compromised server | Partial: it is in pass, but the server can read it |
| Tested restore | A documented periodic test | Done: the three scenarios of 05-08 |
The concrete improvement that follows: restic supports append-only mode on the remote destination, in which the client can write new backups but cannot delete existing ones. With that, an attacker with access to the server cannot destroy the history. It is the measure to ask the external storage provider for, and it goes on the checklist.
A hardening checklist and maintaining the posture
This is the table that gets handed over, aligned with the structure of the CIS Benchmarks and with what you have actually done in the course. The three columns are deliberate: without the evidence column, a checklist is a statement of intent.
| # | Control | Status | Evidence |
|---|---|---|---|
| 1 | Filesystem: noexec,nosuid,nodev on /tmp, /var/tmp, /dev/shm |
Done | findmnt -o TARGET,OPTIONS |
| 2 | Encryption at rest of the backup volume | Done | cryptsetup luksDump, /etc/crypttab |
| 3 | Signed repositories; official updates only | Done | /etc/apt/sources.list.d/*.sources, /etc/apt/keyrings/ |
| 4 | Automatic security updates | Done | unattended-upgrades, -security only |
| 5 | Services pending restart after an update | Done | needrestart -r l with nothing pending; incident closed |
| 6 | Minimal listening surface | Done | ss -tulpn: 22, 5432 internal, 8080 on localhost |
| 7 | Unnecessary services disabled and masked | Done | ModemManager masked and purged |
| 8 | No graphical environment | Done | An Ubuntu Server installation |
| 9 | Firewall with a default-deny policy | Done | ufw status verbose; documented nftables ruleset |
| 10 | Network sysctl hardened |
Done | /etc/sysctl.d/ (06-03) |
| 11 | Kernel sysctl hardened |
Done | /etc/sysctl.d/60-hardening.conf, commented |
| 12 | SSH: no root, no passwords, keys only | Done | sshd -T -C user=...; the Match exception removed |
| 13 | Lockout on failed attempts (IP and account) | Done | fail2ban-client status sshd, faillock --user |
| 14 | Password policy and expiry | Done | pwquality.conf, login.defs, chage -l |
| 15 | A single UID 0; no accounts without a password | Done | The three audit one-liners |
| 16 | Least privilege in sudo |
Done | /etc/sudoers.d/tramontana with Cmnd_Alias |
| 17 | MAC: AppArmor profile in enforce |
Done | aa-status, the tramontana profile |
| 18 | Service with systemd hardening | Done | systemd-analyze security: 1.6 OK |
| 19 | Secrets out of the configuration files | Done | LoadCredentialEncrypted, pass |
| 20 | TLS: material issued, verified and with renewal tested | Done | certbot renew --dry-run, check_certificate.sh |
| 21 | Persistent logging with rotation | Done | /var/log/journal, logrotate.d/tramontana |
| 22 | File integrity watched, database off the machine | Done | tramontana-integrity.timer, the checksum on the laptop |
| 23 | Auditing of access to configuration and identities | Done | auditctl -l, keys tramontana_conf and privileges |
| 24 | 3-2-1 backups, encrypted, with a tested restore | Done | restic snapshots, check_backup.sh, the runbook |
| 25 | Encryption in transit in force | Outstanding (08-01) | Certificate ready; the reverse proxy is missing. 8080 is not exposed |
| 26 | Off-site backup in append-only mode |
Outstanding | Requires a provider decision; it mitigates ransomware |
| 27 | Unexplained authorized_keys2 |
Open | The incident response procedure from 06-04 is under way |
| 28 | Centralised logs off the server | Accepted | Current cost; the first investment when there is budget |
| 29 | NIDS | Accepted | It adds little with a single server (reasoned in 06-04) |
| 30 | Antivirus | Accepted | A server with no third-party files; it consumes limited memory |
| 31 | A second factor on SSH | Accepted | Access is already key-based; to be reviewed if the team grows |
| 32 | Application security (injection, logic) | Out of scope | Development's responsibility |
| 33 | Physical and hypervisor security | Out of scope | The provider's responsibility |
| 34 | A formal security audit | Out of scope | Requires an independent third party |
Rows 25 to 27 are the genuinely important ones, because they are the ones an honest report does not hide. And rows 32 to 34 delimit what an administrator cannot resolve, which is information the management needs to have.
Maintaining the posture
A hardened system degrades on its own: every change, every new package, every "temporary" rule moves it further from the state you documented. Four practices sustain it:
- Periodic review with comparable metrics. Lynis monthly, with its hardening index as a time series alongside the performance baseline from 05-07:
From the 68 in 06-04 to the current 82. What matters is not the number: it is that if it drops, there is an explanation to go and find.$ sudo lynis audit system --quiet | grep 'Hardening index' Hardening index : 82 [################ ] - Change management. Every configuration modification with its prior copy, its
diff -u, its verification and its note in the record. It is the course's convention, and it is the control against the most likely actor in the threat model. - Living documentation in the runbook, off the server: the threat model, this checklist with dates, the inventory of secrets with their last rotation, and the incident response procedure.
- A review of the threat model itself when reality changes: one more server, a new service, a new kind of data that starts being processed. A threat model from two years ago describes a system that no longer exists.
Compliance warning
srv-tramontana processes guests' personal data. The GDPR requires technical and organisational measures appropriate to the risk (art. 32), and several of this lesson's decisions — log retention, the scope of auditing over working people's activity, the timescale for deploying encryption in transit in row 25 — have legal implications that are not an administrator's to decide. In a real environment:
- The security officer must review the threat model and the checklist, and validate the accepted controls.
- The data protection officer must validate the processing, the retention periods and the residual risk assessment.
- And this checklist does not replace a formal audit. It is an honest self-assessment, made by whoever administers the system, and by definition it has the blind spot of the person who built it.
Common Mistakes and Tips
- Hardening with no threat model. It is the cause of half the fragile systems out there: copied measures that break things and protect against nothing that was likely. Half a page of model changes every decision that follows.
- Touching PAM without a root session open. A syntax error in
common-authcan prevent every login, the console included. A root session in another terminal, a prior copy, and a test in a third. - Forgetting
pam_faillock.so authsucc. Without that line the failure counter is never reset, and the lockout arrives days later with no apparent cause. - Forgetting
enforce_for_root = 1inpwquality.conf: the most important account ends up exempt from the password policy. - Putting an AppArmor profile straight into
enforce. It breaks the service and forces you to discover the missing accesses under pressure. Alwayscomplainfirst, exercise it for real, thenenforce. - Applying
noexecto/tmpwithout checking. Some installers extract and run there. Test withmount -o remountand exerciseaptand your scripts before touchingfstab. - Editing
fstabwithoutmount -a. A brokenfstabstops the server booting. It is the safety net from 05-04 and it is still compulsory. - Confusing "the package is updated" with "the vulnerability is fixed". While the process has the old library in memory, it is still exposed.
needrestart -r lafter every update, andlsof | grep DELas confirmation. - Hardening and not verifying.
systemd-analyze securityimproves the score even if you have left the service unable to start. Apply, verify withhealth_check.sh, and then measure. MemoryDenyWriteExecuteon an application with a JIT. Java, Node and .NET will not start. Read what each directive does before copying it.- A checklist with no evidence column. "Firewall configured" says nothing;
ufw status verbosedoes. Without evidence, the checklist is a statement of intent. - A tip on method. Turn this lesson's audits into
audit_hardening.shusinglib/common.sh, with a monthly timer and the principle of silence if all is well. What depends on your memory is not a control.
Exercises
Exercise 1
After applying the hardening drop-in, tramontana.service stops starting:
$ sudo systemctl status tramontana.service --no-pager | head -6
● tramontana.service - Tramontana Bookings
Active: failed (Result: exit-code) since Tue 2026-08-18 16:04:11 CEST
Process: 9214 ExecStart=/opt/tramontana/app/tramontana ... (code=killed, signal=SYS)Describe the diagnostic procedure, identify the most likely cause from the information in the message, and explain how you would narrow down which specific directive causes it without removing them all at once.
Exercise 2
Write the AppArmor profile for the /home/operator/bin/deploy-safe wrapper, which is run via sudo and calls deploy.sh. Reason about what it must permit and — more importantly — what it must deny, and explain which specific attack that profile contains that neither DAC nor the sudoers rule contains.
Exercise 3
Marta asks you for a one-page report for the management: "are we secure?". Write it, drawing on the threat model and the checklist, without unnecessary jargon, honestly saying what is protected, what is not and which decisions need budget or authorisation.
Solutions
Solution 1
The clue is in the message itself: code=killed, signal=SYS. SIGSYS is the signal the kernel sends when a seccomp filter rejects a system call. That points directly at SystemCallFilter, and not at the other directives (which would typically produce EACCES, EPERM or a path failure).
The procedure, from the most informative to the most expensive:
# 1. Confirm the hypothesis and see WHICH call was rejected
$ sudo journalctl -u tramontana.service -n 30 --no-pager | grep -iE 'seccomp|SYS|syscall'
audit: type=1326 audit(1755530651.882:88): auid=4294967295 uid=997 pid=9214
comm="tramontana" exe="/opt/tramontana/releases/3.2.1/tramontana"
syscall=318 compat=0 ip=0x7f2a1c4b3e2a code=0x80000000
# 2. Translate the call number into its name
$ ausyscall 318
getrandomgetrandom is the call for obtaining randomness from the kernel — the application uses it for TLS and for session identifiers. It is in @system-service, but I excluded it by adding SystemCallFilter=~@privileged @resources @obsolete, and getrandom belongs to the @resources set in some versions of systemd. The second line, which looked like an improvement, is the one that breaks the service.
How to narrow it down without removing everything, which is the method part of the exercise. The key is that drop-ins can be stacked and overridden separately:
# a) A temporary drop-in that only neutralises the suspect directive.
# The empty line RESETS the accumulated list; then only the minimum is set.
$ sudo mkdir -p /etc/systemd/system/tramontana.service.d
$ sudo tee /etc/systemd/system/tramontana.service.d/99-diagnostic.conf >/dev/null <<'EOF'
[Service]
SystemCallFilter=
SystemCallFilter=@system-service
EOF
$ sudo systemctl daemon-reload && sudo systemctl restart tramontana.service
$ ~/scripts/health_check.sh; echo "status: $?"
status: 0It starts. That confirms the problem was the exclusion, not the base set. Now you refine instead of giving up on the filter:
# b) Recover the useful restriction, giving back only what is necessary
$ sudo tee /etc/systemd/system/tramontana.service.d/99-diagnostic.conf >/dev/null <<'EOF'
[Service]
SystemCallFilter=
SystemCallFilter=@system-service
SystemCallFilter=~@privileged @obsolete
SystemCallFilter=getrandom
EOF
$ sudo systemctl daemon-reload && sudo systemctl restart tramontana.service
$ ~/scripts/health_check.sh && systemd-analyze security tramontana.service | tail -1
→ Overall exposure level for tramontana.service: 1.7 OK 🙂~@privileged @obsolete is kept — which is where nearly all the defensive value lies — @resources is abandoned, and getrandom is added explicitly. Exposure 1.7 instead of 1.6, with the service running: worse hardening on paper and infinitely better in practice, because the 1.6 one did not start.
A general method applicable to any hardening failure:
- Read the signal or the error code:
SIGSYS→ seccomp;EPERMon a file →ProtectSystem/ReadWritePaths;EACCESwithapparmor="DENIED"→ AppArmor; a failure to map memory →MemoryDenyWriteExecute. - Find the concrete evidence in the journal (
syscall=,denied_mask=, the path). - Neutralise one directive with a drop-in with a higher numeric prefix, never by editing the original.
- Verify that it starts, refine towards the necessary minimum, and measure again.
- Consolidate the drop-in with a definitive name and write down the reason for the exception — just like the commented-out
unprivileged_userns_clonein thesysctlfile.
Solution 2
# /etc/apparmor.d/home.operator.bin.deploy-safe
abi <abi/4.0>,
include <tunables/global>
/home/operator/bin/deploy-safe {
include <abstractions/base>
include <abstractions/bash>
# The wrapper and the script it invokes
/home/operator/bin/deploy-safe r,
/home/operator/scripts/deploy.sh rix,
/home/operator/scripts/lib/common.sh r,
/usr/bin/bash rix,
# The specific utilities the script needs, ENUMERATED
/usr/bin/{tar,curl,ln,sha256sum,systemctl,flock,mktemp,rm,mv,date,logger} rix,
# Source of the release package: read only
/srv/tramontana/backups/outgoing/*.tar.gz r,
/srv/tramontana/backups/outgoing/*.sha256 r,
# Deployment destination: writing confined to releases and the link
/opt/tramontana/releases/ rw,
/opt/tramontana/releases/** rw,
/opt/tramontana/app rw,
/opt/tramontana/HISTORY rw,
# Log and lock
/var/log/tramontana/deploy.log rw,
/var/lock/tramontana-deploy.lock rwk,
/tmp/ r,
/tmp/** rw,
# Query the service's state after the deployment
/run/systemd/private rw,
# Explicitly DENIED, so that it is on record and gets logged:
deny /etc/shadow rwklx,
deny /etc/tramontana/secrets/** rwklx,
deny /srv/tramontana/backups/restic/** rwklx,
deny /home/operator/.ssh/** rwklx,
deny /home/operator/.password-store/** rwklx,
deny /root/** rwklx,
deny /usr/bin/{nc,ncat,socat,ssh,scp,python3,perl} x,
}What it permits: reading the release package, writing under releases/, changing the app link, writing its log and its lock, and talking to systemd to restart the service. Nothing else.
What it denies, and which attack each denial contains:
| Denial | The attack it contains |
|---|---|
/etc/tramontana/secrets/** |
The process runs as root via sudo, so DAC would let it read the encrypted credential. AppArmor does not |
/srv/tramontana/backups/restic/** |
A deployment has no reason whatsoever to touch the backups. It contains the deletion of backups by ransomware |
/home/operator/.password-store/** |
Stops the wrapper reaching the source of truth for the secrets |
/home/operator/.ssh/** |
Blocks adding an authorised key — the persistence mechanism AIDE detected in 06-04 |
x on nc, socat, python3, perl |
Blocks the reverse shell, which is the first step after obtaining privileged execution |
And now the substance of the exercise: what this profile contains that DAC and sudoers do not.
The sudoers rule lets operator run deploy-safe as root. From that moment on, DAC offers no protection at all: root can read /etc/shadow, can read the database credential, can delete the backups, can add itself an SSH key and can launch nc to open a shell outwards. The wrapper was written precisely to avoid granting full root, but the containment depends entirely on the content of the script being correct and staying correct.
That is where the gap is. If deploy.sh has a flaw — an unquoted variable that allows a command to be injected, a tar that extracts a file with ../ in its path — the attacker executes code as root through an authorised route. Neither sudo nor the Unix permissions stop it: from their point of view the execution is legitimate.
The AppArmor profile is the layer that does stop it, because the control does not depend on the user but on the program: whatever code ends up running inside this profile, it will not be able to read the secrets, will not be able to touch the backups, will not be able to write to authorized_keys and will not be able to launch an interpreter to escape. The damage stays confined to the deployment, which is exactly what was intended when the wrapper was written and what until now was only an intention.
That is defence in depth in its most concrete form: sudoers limits who and which command; AppArmor limits what that command can do, even when the command misbehaves.
Solution 3
Security report —
srv-tramontana/ Tramontana Bookings To: Management · From: Systems Operations · 18 August 2026In one sentence. The server is reasonably protected against the threats that genuinely affect it, with three outstanding matters that I set out at the end, one of which requires a decision from management.
What we are protecting, in order of importance. First, our guests' personal data: names, dates of stay and amounts. A leak would be irreversible, would harm third parties and has legal consequences. Second, the availability of the booking service, whose cost is economic and bounded in time.
Who we are protecting ourselves from, and what we have done.
- Automated attacks from the Internet, which are constant. The server accepts connections through two doors only: the administration one — which no longer accepts passwords, only cryptographic keys — and the website's. Everything else is closed by default, and anybody who persists in trying passwords is blocked automatically. In the last month 47 attempts from a single source have been blocked.
- Theft or leakage of a password. The system's credentials are no longer written in any readable file: they are encrypted and only the program that needs them can use them. We also have a written procedure for changing them without interrupting the service.
- Our own mistakes, which statistically are the most likely risk. Every modification is made with a prior copy, is checked before being applied and is recorded. An independent system checks every day that no critical file has changed without explanation.
- Misuse of legitimate access. Each person has only the permissions their job requires, and accesses to the configuration are recorded.
What we do if something fails. We have encrypted backups, with the restore tested — not merely configured — capable of recovering the service in 8 hours with a maximum loss of 4 hours of bookings, timescales agreed with Operations. And a written incident response procedure, kept off the server.
The three outstanding matters.
- Web traffic is not yet encrypted. The certificate has been issued and verified, but the component that will present it still has to be deployed, and that is scheduled for the next phase. In the meantime the service is not reachable from the Internet, so the exposure is limited to our internal network. No decision required: it is planned.
- An unexplained finding. Our monitoring detected an access configuration file that nobody created. It is under investigation following the established procedure. I will report the outcome; if access to personal data were confirmed, there is a legal obligation to notify it within 72 hours.
- An off-site backup protected against deletion. Today, anybody with control of the server could destroy the backups. There is a form of storage that does not allow what has already been stored to be deleted, and it is the best available defence against a data-hostage attack. Requires a management decision: it means contracting external storage.
What is not in our hands. Three things fall outside what systems administration can resolve, and it is worth putting them on record: the internal security of the application itself, which is development's responsibility; physical and infrastructure-provider security; and an attacker with resources far greater than those of a company of our size.
And a necessary caveat. This report is a self-assessment made by whoever administers the system, and it therefore has the blind spot of the person who built it. It does not replace an independent audit. I also recommend that the processing of guests' data and the retention periods for logs be reviewed by the data protection officer.
In summary: the work is done, measured and documented. The only decision I need from management is the one in point 3.
Conclusion
srv-tramontana now has a security posture, and that is qualitatively different from having security measures. There is a written threat model that says what we protect, from whom and what we assume to be out of scope, and every decision in this lesson is justified by it. The attack surface has been reduced: the application listens only locally, and what is not used is masked and purged. Authentication goes through PAM with a password policy, per-account lockout and resource limits — and you tracked down the Match User luis that had been contradicting your own configuration for weeks. AppArmor contains the application in enforce, with a profile that prevents launching a shell or reading the backups even if the process is compromised. The hardening sysctl settings are applied and commented, including the line you left disabled with the reason written beside it. /tmp has no execute permission, checked before it was made persistent. And systemd-analyze security has gone from 3.4 to 1.6 with the service verified and working, which is the only improvement that counts.
The three incidents are closed: the 47 attempts from 203.0.113.44 with fail2ban, the db_password in the clear with systemd-creds and pass, and today the vulnerable libssl that was still loaded in memory — because an updated package is not a fixed vulnerability until the processes know about it. What remains is a 34-row checklist with its evidence column, three open matters stated without decoration and a report for the management that does not promise what it cannot deliver. The Lynis index has risen from 68 to 82, and the important thing about that number is not its value but that it is now a time series somebody watches.
And with that, Module 6 closes: you configured the network persistently with netplan try's automatic rollback; you hardened SSH with ed25519 keys, no root and no passwords, and learned to read sshd -T -C to find out what configuration really applies to somebody; you raised an allowlist firewall over nftables and learned the order it is applied in so as not to lock yourself out; you set up intrusion detection with AIDE and auditd, with the database protected off the server, and an incident response procedure with its uncomfortable rule; you got the secrets out of the configuration files and prepared the TLS cryptographic material with renewal tested and expiry monitored; and today you have brought all of that together into a coherent, documented posture.
So far you have operated one server, and you have operated it by hand. You know how to use it, administer it and defend it, but there are two things you still cannot do: look inside and multiply it. In Module 7: Advanced Topics you do both. You will see the complete boot process — from UEFI to GRUB, to the initramfs and to systemd — and learn to recover a system that will not boot, which is the skill that separates the people who reinstall from the people who fix. You will learn advanced diagnosis with strace, perf and eBPF to answer "what exactly is this process doing?" when the counters are not enough (and you will discover that the ptrace_scope you have just set is part of the problem statement). You will tune the kernel with sysctl, this time for performance. And then the multiplication begins: virtualisation with KVM and libvirt, containers with Docker and why they are isolated processes and not small machines, automation with Ansible — which will turn all the manual work of modules 5 and 6 into versioned, reproducible code, and the 8-hour RTO into considerably less — and finally high availability and load balancing, where srv-tramontana stops being a single point of failure. Update your VM snapshot, keep the runbook off the machine, and I will see you in Module 7.
Linux Course: From Beginner to System Administrator
Module 1: Introduction to Linux
- What Is Linux?
- History of Linux
- Linux Distributions
- Installing Linux
- First Contact with the System
- The Linux File System Structure
Module 2: Basic Linux Commands
- Introduction to the Command Line
- Getting Help and System Documentation
- Navigating the File System
- File and Directory Operations
- Viewing and Editing Files
- Hard and Symbolic Links
- File Permissions and Ownership
Module 3: Advanced Command-Line Skills
- The Shell Environment: Variables, Aliases and History
- Using Wildcards and Regular Expressions
- Searching Files and Content: find, locate and grep
- Pipes and Redirection
- Text Processing: cut, sort, uniq, sed and awk
- Process Management
- Scheduling Tasks with Cron
- Networking Commands
Module 4: Shell Scripting
- Introduction to Shell Scripting
- Variables and Data Types
- Script Input, Output and Arguments
- Control Structures
- Functions and Libraries
- Debugging and Error Handling
- Production Scripts: Best Practices
Module 5: System Administration
- User and Group Management
- sudo and Special Permissions
- Package Management
- Disk Management
- systemd and Service Management
- System Logs: journald and syslog
- System Monitoring and Performance Tuning
- Backup and Restore
Module 6: Networking and Security
- Network Configuration
- SSH and Remote Access
- Firewalls and Perimeter Security
- Intrusion Detection Systems
- Secrets Management and TLS Certificates
- Securing Linux Systems
Module 7: Advanced Topics
- The Boot Process and System Recovery
- Advanced Diagnostics: strace, perf and eBPF
- Linux Kernel Tuning
- Virtualization with Linux
- Linux Containers and Docker
- Automation with Ansible
- High Availability and Load Balancing
