The three previous lessons have built prevention: an access control model, well-managed identities, a confined service and a hardened system. All of that reduces the probability of something bad happening and limits the damage if it does. But none of those measures answers the question that really keeps anyone running a production server awake: how do I know whether something has happened?

Imagine the concrete scenario we will solve at the end of this lesson. It is 09:00 and monitoring raised an alert in the small hours: between 03:12 and 03:40 there was a spike in writes to /var/lib/meteora twenty times the norm for that hour, and right now there is a running process whose name nobody on the team recognizes. What do you do? Kill it? Reboot the server? Disconnect the network? Look at the logs first, or at the process? Each of those decisions, taken in the wrong order, destroys information that will not come back, and some of them make the incident worse instead of containing it.

This lesson gives you the method. First, what gets logged on a Linux system and where: syslog with its facilities and priorities, and journald with its filters — the ones you will actually use. Then, how an application's logging is designed for something like meteo-api: what every event must contain and, above all, what must never appear there, with the corresponding GDPR warning. Then rotation and retention, the kernel's auditd subsystem with its real performance cost, and the most uncomfortable problem of all: the integrity of a log written by a system the attacker controls. We go on with host change detection — AIDE, rootkits, concrete indicators — and finish with the incident response cycle applied step by step to the 03:12 case, with evidence preservation and its legal implications.

1. Why without logging there is no detection and no response
2. syslog: facilities, priorities and /var/log/
3. journald and journalctl in practice
4. Designing an application's logging
5. Rotation and retention with logrotate
6. The kernel's audit subsystem: auditd
7. Log integrity against an attacker with privileges
8. Detecting changes and intruders on the host
9. The incident response cycle
10. Case study: the 03:12 spike
11. Evidence preservation
12. Which preventive measures each incident leaves behind

Security controls fall into three families, and it is worth seeing where each thing we have learned fits:

A system with preventive controls and no detective controls has a structural problem: when prevention fails, nobody finds out. And prevention always fails at some point, because software has bugs, configurations decay and people make mistakes.

Without logs, four things become impossible, not merely difficult:

• Detecting. An attacker who breaks nothing visible can stay for months. The median undetected dwell time in real incidents is measured in weeks, not hours.
• Investigating. With no trail you cannot answer the questions that matter: what got in, when, through where, what it touched and what it took.
• Containing sensibly. Without knowing the scope, the only option is to shut everything down, which is usually disproportionate and sometimes destroys the evidence.
• Learning. An incident from which no concrete preventive measure is drawn will happen again.

Hence the working rule of this lesson: logging is not a by-product of the system, it is a security control and it is designed as such, with its content, its retention, its protection and its verification.

syslog is the classic UNIX mechanism: a daemon receives messages from every program and classifies them along two dimensions. It is still relevant because it is the industry's common language: routers, firewalls, storage arrays and central collectors all speak syslog.

The ones that will matter most to you: auth and authpriv collect everything to do with authentication and elevation — logins, sudo, PAM — and authpriv is the variant for anything that may contain sensitive data, which is why its file has more restrictive permissions. And local0–local7 are reserved for your own applications: meteo-api can use local3, for instance, keeping it separate from everything else without touching the rest of the system.

How to read that configuration: each line is a selector (facility.priority) followed by a destination. local3.* sends everything from the application to its own file. *.emerg notifies all logged-in users of emergencies. And the last line is the important one for section 7: @@ means sending over TCP with TLS to a remote collector — with a single @ it would be UDP, with no delivery guarantee — so that authentication events also exist off the machine. That detail is what separates a log that is useful in an investigation from one that is not.

On a current Debian, the main log is the systemd journal: a binary, indexed store in which every entry carries structured fields as well as the text — unit, PID, UID, executable, priority, boot identifier — which is why it can be queried like a small database.

The decisive property is the third one: journald adds the metadata itself, taking it from the system and not from the text of the message. A process cannot lie about its PID, its UID or its unit, because it is not the one writing them. In syslog, anyone who can write to the socket can claim to be whoever they like.

The ones that make the difference in an investigation. --since and --until narrow the time window, which is always the first step: you do not read "the log", you read the relevant half hour. -b -1 queries the previous boot, essential when the server has been rebooted — or someone rebooted it — and you need to see what happened just before. The per-field filters (_UID=, _COMM=, _PID=) are the real advantage of the indexed format: journalctl _UID=990 --since "-24h" gives you everything the service's identity did in a day, without depending on the message text containing the right word. And -o json-pretty exposes every field, which is what you need to feed a collector or to automate a check.

The persistence check is not optional. Without /var/log/journal, the journal lives in /run, which is tmpfs (04-03): it is lost entirely on every reboot. An attacker who reboots the machine erases the log without touching a single file, and you are left investigating an incident of which nothing remains. It is one of the first things to verify on a freshly installed server.

And the three retention limits, which have to be set deliberately:

MaxRetentionSec is the security decision in this block: it defines how far back you can investigate. If a compromise is detected six weeks later — as is usual — and your retention is seven days, the logs from the moment of entry no longer exist and you will never know how they got in.

A useful log does not happen by itself: it is designed. These are the fields every meteo-api event must carry:

Three decisions in this code. The structured format (JSON) allows querying and aggregating without depending on fragile regular expressions; a collector can filter by event without understanding the text. The correlation identifier is generated when the request arrives and accompanies every event it causes, including those of ingestor and aggregator if it is propagated: it is what lets you reconstruct one request across three services. And the reason for the failure is always logged, because "denied" with no reason is of no use whatsoever in an investigation.

The case of failed passwords deserves a pause. Logging the failed attempt "for investigation" seems harmless, and it is one of the worst possible practices: most failures are typing errors on the right password, so the log ends up containing, in the clear and with the user name next to it, your users' real credentials. And that log is read by the whole adm group, travels to the central collector and goes into the backups.

With personal data, the operational rule is log identifiers, not people: client=CLI-4471 instead of the name and the email address, and a separate table — with its own access control — that translates the identifier when it is genuinely needed.

GDPR and compliance warning. An IP address, a user identifier or an access history are personal data under the GDPR. Recording them requires a legal basis, information to the data subjects, minimization, a defined and justified retention period, access control and, where applicable, an impact assessment. Log retention also routinely conflicts with the right to erasure, and there are sector obligations that impose minimum periods. Define the logging and retention policy together with the compliance officer or legal counsel, in writing, before deploying it, and do not improvise it in the middle of an incident.

Without rotation, a log grows until it fills the disk, and a full /var/log stops services — including logging itself, so you lose the trail exactly when you need it most.

The three directives that cause trouble if misunderstood. create 0640 meteora adm is essential: without it, the new file inherits the umask of the rotation process — which runs as root — and may end up with the wrong permissions or with owner root, and then the service can no longer write to its own log. postrotate with reload exists for a reason you already know from 04-04: when logrotate renames the file, the process keeps writing to the same inode through its open descriptor, so its messages go to the rotated file and the new one stays empty; it has to be told to reopen. The alternative is copytruncate, which copies and truncates in place, but loses the lines written between the copy and the truncation, so it is only used when there is no way to signal the process. And delaycompress avoids compressing a file that may still be open.

The retention trade-off is worked out with concrete numbers. meteo-api generates about 40 MB of log a day; ninety days uncompressed would be 3.6 GB, and compressed comes to around 400 MB. The question that decides the value is not how much space it takes, but how far back you need to be able to investigate:

Ninety days is a reasonable starting point for meteo-01: it covers the realistic case of late detection without accumulating personal data for a year. But it is a decision that must be validated with compliance, because there may be mandatory minimum periods or maximums imposed by minimization.

journald records what programs decide to tell you. auditd records what happens in the kernel, whether the program likes it or not: which process opened which file, which system call was executed, who changed a permission. It is the difference between the suspect's testimony and the camera footage.

Each block answers a different need. The -w rules watch paths: notice that secrets.conf also carries r, because in a key file reading it is already the relevant event, not just modifying it. The system call rules are more precise: the first records every execve run with root privileges by someone who logged in as a normal user — auid is the audit identifier, which does not change with su or sudo and therefore lets you attribute the action to the real person; the following ones cover mounting, module loading and permission changes in the data tree. And -e 2 makes the rules immutable: from then on not even root can modify them without rebooting the machine, and a reboot is itself a highly visible event.

ausearch searches by key, by audit user, by time range or by process, and -i is practically mandatory because it translates the numbers into readable names. aureport produces summaries: the failed authentication one is among the most useful for a daily review.

The performance cost is real and has to be sized. Every audited event means work in the kernel and a write to disk, and a badly thought-out rule can degrade the system noticeably. The dangerous rule par excellence is auditing all reads and writes of an active tree: the ingestor writes 720,000 readings a day to /var/lib/meteora/readings, so a -S read -S write rule on that directory would generate millions of events a day, fill the disk in hours and add latency to every operation. The correct criterion is to audit the rare, not the frequent: configuration changes, access to secrets, privilege elevations, module loading, mounts. And check the volume with aureport --summary during the first few days to tune it.

We come to the most uncomfortable problem, and the one that has to be clear before anything else:

An attacker with root on meteo-01 can modify or delete any local log. They can edit /var/log/auth.log, purge the journal, stop auditd, remove a chattr +a with CAP_LINUX_IMMUTABLE and rewrite history. A local log is never sufficient proof of anything.

That does not mean local defenses are useless; it means you have to understand what each one contributes.

The remote shipping configuration has four important decisions. TCP with TLS guarantees delivery and confidentiality, unlike UDP, which loses messages silently exactly when there is saturation — which is when incidents happen. StreamDriverAuthMode="x509/name" makes the sender verify the collector's identity, so nobody can impersonate it and absorb your logs. The on-disk queue with saveOnShutdown avoids losing events if the collector is unavailable for a while. And resumeRetryCount="-1" retries indefinitely.

And the three properties the collector must have for all of this to be worth anything:

Separate credentials. meteo-01 must be able to send events and not be able to read or delete them. If the server's certificate allowed administering the collector, an attacker with root on meteo-01 would delete the remote copy as well, and we would have gained nothing.

Immediate, append-only writing at the destination. The value of remote shipping is that the event leaves at the moment, before anyone can delete it; batched shipping every fifteen minutes leaves a perfectly exploitable fifteen-minute window.

Independent retention. The period is set by the collector, not by the originating server.

The underlying idea, worth stating explicitly: the evidence must get out of the attacker's reach as soon as possible. It is the same reasoning as the immutable backups of 05-03 and, incidentally, the reason why in section 11 the memory capture is taken before anything is touched.

AIDE computes a reference baseline with the checksums, permissions, owners and timestamps of every file you tell it about, and then compares against it periodically. It detects exactly what an attacker needs to do in order to persist: modify a binary, add a systemd unit, touch the configuration.

Two decisions are what make AIDE work or turn it into useless noise. The rules by file type: for a binary you watch everything, but for a log that grows every second you cannot watch the size or the checksum, only permissions and owner — hence the LogFile rule with S, which tolerates growth but detects a truncation. And the exclusions: /var/lib/meteora/readings changes constantly by design, so including it would generate thousands of differences a day and nobody would ever read the report again.

And the critical point: the reference baseline must be stored off the machine, or at least on read-only media. If it lives in /var/lib/aide/aide.db and the attacker has root, they update it after making their changes and AIDE will report that everything is in order. The same goes for the setuid and capability reference lists of 05-02.

These tools compare binaries against known checksums and look for common patterns. They are useful and they have a limit you must know: if the compromise is in the kernel, the system's answers are not trustworthy, including the ones these tools receive. They are good for the common case, not for the sophisticated one.

The indicators you look for by hand, with the command and why it is suspicious:

The first one deserves an explanation because it is the most revealing of all: (deleted) in /proc/<pid>/exe means the program is running but its file no longer exists on disk. It is a classic evasion technique — the binary is executed and deleted immediately afterwards, so that no file system analysis finds it — and although it has legitimate explanations (a binary updated while the old process is still alive, exactly the case from 04-02), on a stable server it is an anomaly that has to be explained. And it has a lovely forensic consequence that we will use in section 11: the file still exists as long as the process lives, because its link count has not reached zero, and it can be recovered by copying /proc/<pid>/exe.

A single server produces events; an organization produces millions, spread across servers, firewalls, applications and cloud services. Centralized correlation — what is generically known as a SIEM, without getting into products — does three things no single host can do on its own: it normalizes different formats into a common schema, it correlates events from different sources — "a successful SSH login from an IP the firewall saw scanning ten minutes ago" — and it alerts on defined patterns. It requires the remote shipping of section 7 and event naming discipline like that of section 4; without those two things, it is an expensive text store nobody queries.

An incident is managed with a method, not by improvising. The six phases are the same in every reference framework:

The two mistakes that do the most damage belong to phase 3, and it is worth understanding them before the case study. Shutting the server down destroys all the volatile memory: processes, connections, encryption keys in RAM and the contents of deleted binaries that only exist through /proc. And just killing the suspicious process removes the evidence and solves nothing, because if there is a persistence mechanism — a scheduled job, a systemd unit, an SSH key — the process will be back in minutes and you will have lost the chance to observe it.

Situation. It is 09:00 on 1 September 2026. Monitoring flagged a spike in writes to /var/lib/meteora between 03:12 and 03:40, twenty times the norm for that hour. There is a process called kworkerd that nobody on the team recognizes.

How to read what you get, and why in this order. The snapshot of the volatile state goes first because of the order of volatility in the next section: processes and connections vanish as soon as anything changes, whereas the files will still be there in an hour's time. The (deleted) in /proc/<PID>/exe confirms the evasion technique of section 8, and the PPid: 1 indicates that the parent process has already died, which usually means it was launched and deliberately abandoned. The Uid: 990 is the most valuable piece of information in the whole block: the process runs as meteora, not as root, so the vector is almost certainly meteo-api or the ingestor, and the confinement of 05-03 tells us straight away how far it could have got. And the copy of the binary from /proc has to be made right now: if the process dies, the inode is freed and the binary is gone forever.

Findings of the case. The process runs as meteora from a deleted binary in /var/tmp/.cache/. auditd shows that at 03:11:58 there was an execve from meteo-api's process tree, one minute before the spike. There are no events under meteora_secrets, so it never got to read the API keys. There are no events under escalation, and NoNewPrivileges=yes explains why. And aide --check flags no changes in /usr/bin or in /etc: the confinement of section 9 of 05-03 prevented writing outside the declared paths. The persistence is limited to a cron entry belonging to the user meteora that relaunches the binary every hour.

The four decisions, with their reasons. Isolating the network instead of shutting down stops exfiltration and remote command while preserving the whole volatile state, which is the most valuable evidence and the first to be lost. kill -STOP instead of kill -9: the process stops running but continues to exist, with its memory, its open descriptors and its /proc/<PID>/exe intact and available for analysis. Cutting off the persistence before anything else, because otherwise it will be back in less than an hour. And copying the data with -a --preserve=all to keep permissions, ACLs and timestamps (04-06), which are part of the evidence.

Eradication. Deleting the binary is not enough: you have to find how it got in. The execve at 03:11:58 from meteo-api's tree points to the application; with the correlation identifier of section 4 you locate the specific request that caused it and identify the bug. It is fixed, the dependencies are updated, and every credential the process could read is rotated — even though auditd says it did not read them, because the absence of an event is weaker proof than its presence.

Recovery. Restore the affected data from the backup (05-03) verifying the checksums, put the service back into production, and maintain heightened monitoring for weeks: specific alerts on execve by meteora, on out-of-hours writes and on new outbound connections. If there had been any indication of kernel compromise or of root access, the only defensible option would be to reinstall from scratch, because on a system with a compromised kernel you cannot trust anything the system says about itself.

Lessons learned. A meeting with no hunt for culprits, with a table of concrete measures, each with an owner and a date. For this case: noexec on /var/tmp (which would have prevented running the binary), SystemCallFilter without execve in meteo-api's unit (which would have prevented launching it), a review of the request parsing code, an automatic alert on execve with _UID=990, and cron disabled for service accounts.

If the incident may have legal consequences — a criminal complaint, insurance, an employment claim, notification to the data protection authority — the way the evidence is collected determines whether it will be worth anything.

The order of volatility dictates the collection sequence, from the most ephemeral to the most durable:

Why not shut down without thinking. A reboot destroys levels 1 to 5 of that table: all the memory, the active connections, the encryption keys that existed only in RAM, the contents of /tmp and /dev/shm — including /dev/shm/meteora-cache — and the deleted binaries that were only reachable through /proc. It is, by a wide margin, the fastest way to lose half the investigation. Shutting down is only justified if the ongoing damage outweighs the value of the evidence; in most cases, isolating the network contains things just as well and preserves everything.

The checksum is what gives the image its value: it is computed when copying and computed again afterwards, and if it matches it proves the image has not been altered since it was taken. The analysis is always done on a working copy, mounted read-only, never on the original or on the affected system.

Chain of custody. This is the documentary record of who has had the evidence at every moment, and without it a technically impeccable piece of proof can be inadmissible. It must record, for each item: what it is and where it came from, who obtained it and when (with time zone), its checksum, where it is stored, and every transfer with date, people and reason. With restricted access and no gaps.

Legal warning, and it is an important one. A security incident can carry formal obligations with deadlines: under the GDPR, a breach affecting personal data requires notifying the supervisory authority within a very short period, and in certain cases the affected individuals as well. There may also be sector, contractual and insurance obligations. As soon as a breach involving personal data is suspected, involve legal counsel and the data protection officer immediately, and assess with them whether to report the matter to the competent authorities. Do not delete anything, do not negotiate with an attacker on your own, and do not make conclusions public before they are confirmed. Decisions about notification, evidence preservation and communication are not technical decisions.

An incident that produces no concrete measures is an incident that will happen again. This is the translation of the 03:12 case, and it serves as a model for any other:

Notice the pattern: most of the measures are configuration directives you already knew about. The difference between knowing them and having them applied is exactly what separates a contained incident from a complete compromise.

Not checking that the journal is persistent. Without /var/log/journal, journald stores in tmpfs and everything is lost on reboot. An attacker who reboots erases the log without touching a file. Verify it on every new server.

Logging passwords or tokens "for debugging". Password failures are usually typos on the right one, so the log ends up containing real credentials in the clear, readable by the adm group, replicated to the collector and present in the backups.

Retaining seven days. A compromise is detected weeks later. With seven days of retention, the logs from the moment of entry no longer exist and you will never know how they got in.

Trusting the local log after a compromise. An attacker with root modifies it. Send the events to a collector with different credentials and do it at the moment, not in batches.

Auditing too much with auditd. A rule on reads and writes of /var/lib/meteora would generate millions of events a day, fill the disk and add latency. Audit the rare, not the frequent, and measure with aureport --summary.

Shutting the server down when an incident is detected. It destroys the memory, the connections, /tmp and the deleted binaries reachable only through /proc. Isolate the network instead.

Killing the suspicious process. It removes the evidence and solves nothing: if there is persistence, it comes back in minutes. Use kill -STOP and look for the persistence mechanism.

Keeping AIDE's reference baseline on the machine itself. An attacker with root regenerates it after their changes and the report will come out clean. The same goes for the setuid and capability lists.

Forgetting create in logrotate. The new file may be born with the wrong owner or permissions, and the service can no longer write its own log. And without postrotate with reload, the process keeps writing to the old inode.

Tip: rehearse the response before you need it. An annual two-hour drill — "an unknown process shows up, what do we do and in what order?" — reveals that the phone numbers are missing, that nobody knows where the backups are and that the retention is seven days. Discovering that in a drill costs a morning; discovering it in a real incident costs vastly more.

On a machine of your own or a virtual one: (a) check whether the journal is persistent and what its real retention is, and fix it if appropriate; (b) locate the last five failed authentication attempts and the last five uses of sudo, stating the command used in each case; (c) write three journalctl queries that use per-field filters instead of grep, and explain the advantage of each one; (d) calculate how much space it would take to retain 90 days of your current logs and decide on a justified retention policy, stating which aspects you would consult with compliance.

Design the service's complete logging system: (a) the events meteo-api must record, with their fields, in structured format, and at least three that it must never record, with the reason; (b) the logrotate file for /var/log/meteora/, justifying each directive; (c) five auditd rules for meteo-01, explaining what each one detects and why you have not included a rule on writes to /var/lib/meteora/readings; and (d) the scheme for protecting log integrity, explaining what each layer contributes and what it does not contribute against an attacker who has obtained root.

At 08:15 you detect: meteo-api restarting in a loop every few minutes; /var/lib/meteora at 98% full when yesterday it was at 60%; a file /var/lib/meteora/readings/README_RECOVER.txt; and the .dat files of the last three days with the extension .dat.locked. Write the complete action plan by phases, with the exact commands in the right order and the justification for every decision. State explicitly: what you must not do and why; which evidence you preserve and in what order; when and why you involve legal counsel; and the five preventive measures this incident would leave behind.

Solution 1

(a) The real retention is not the configured one but the one that results from the space caps: if SystemMaxUse is reached before MaxRetentionSec, journald deletes the old entries and the effective window is shorter than you think. That is why the valid check is to look at the date of the oldest entry, not at the configuration file.

(c) The advantage of per-field filters over grep is twofold. They are precise: _UID=990 selects the events the kernel attributes to that UID, whereas grep 990 would also match a 990 appearing anywhere in a message. And they are unforgeable: the underscore fields are added by journald taking them from the system, not from the sender's text, so a process cannot lie about its UID or its unit, which it certainly can do with the message body.

(d) The calculation is journalctl --disk-usage divided by the current retention in days, multiplied by 90. The decision has to balance the investigation window — 90 days covers the realistic case of late detection — against data minimization, because the logs contain IP addresses and user identifiers, which are personal data. With compliance you must discuss: the legal basis for the processing, the maximum justifiable period, whether there are mandatory minimum periods in the sector, how access and erasure rights over the logs are handled, and who may read them.

Solution 2

(a) meteo-api events:

Always log: authentications (successful and failed, with the reason), data accesses with the resource and the volume, configuration changes, starts and stops, and errors with enough context to reproduce them.

Never, with the reason: passwords, failed ones included, because most failures are typos on the correct password and the log would end up containing real credentials in the clear; tokens and API keys, because whoever reads the log can reuse them directly — and the log is read by the adm group, travels to the collector and goes into the backups; and unnecessary personal data such as the client's name and email address, which are replaced by the identifier CLI-4471 plus a translation table with its own access control, applying the minimization principle.

(b) The logrotate file is the one from section 5. Justification per directive: daily + rotate 90 sets the investigation window at 90 days; compress reduces the space taken by .log files by roughly 10:1; delaycompress avoids compressing a file that may still be open; create 0640 meteora adm is essential because without it the new file may be born with owner root and the service would no longer be able to write; and postrotate with reload exists because after renaming, the process keeps writing to the same inode through its open descriptor (04-04), so without signalling it the new file stays empty.

(c) Five rules and what each one detects:

(1) detects access to the secrets, and carries r because in a key file reading it is already the event. (2) detects the granting of new privileges. (3) is the one that would have detected the 03:12 incident as it happened, because meteo-api must not execute anything. (4) detects the step before a kernel rootkit. (5) detects one of the most common forms of persistence.

I do not include a rule on writes to /var/lib/meteora/readings because the ingestor writes 720,000 readings a day: it would generate millions of events a day, fill the disk in hours and add latency to the service's critical path, with practically no detection value, since those writes are normal operation. The criterion is to audit the rare, not the frequent.

(d) Log integrity:

The conclusion that organizes the table: only the last row really protects against an attacker with root, and the earlier ones have real value against accidents, against unprivileged users and for detecting tampering after the fact.

Solution 3

What you must NOT do, and why. Do not shut down or reboot: it would destroy the memory, the connections, /tmp and /dev/shm, and with them possible encryption keys in RAM that in some ransomware cases allow the data to be recovered without paying. Do not delete the encrypted files: they are evidence and are sometimes recoverable. Do not restore the backup immediately onto the same system: if the attacker is still inside, they will encrypt the restored copy too. Do not pay or negotiate on your own initiative: that is a management decision with legal implications. And do not kill the process before documenting it.

Justification. The disk at 98% explains the restart loop: meteo-api cannot write and Restart=on-failure relaunches it; it is a symptom, not the cause. The ransom note and the .locked extensions confirm ransomware. The network is isolated to cut off communication with the attacker while preserving the volatile state; the service is stopped, since it no longer works and only adds noise; the process is frozen rather than killed so that it can be analyzed; and the volume is remounted read-only so that nothing else gets encrypted. The evidence is collected in order of volatility: memory and processes, then network, then temporary file systems, and finally the disk.

Legal counsel: immediately, and in parallel with the analysis. Ransomware almost certainly involves a personal data breach, with a possible obligation to notify the supervisory authority within a very short period, and perhaps the affected customers as well. On top of that, you have to assess reporting the matter to the competent authorities, contractual obligations towards customers and communication with the insurer. None of that is a technical decision and all of it has deadlines.

Recovery: rebuild the server from scratch — not clean up the existing one, because you cannot prove no persistence remains — restore from a backup predating the compromise verifying the checksums, apply the hardening of 05-03 before exposing it again, rotate every credential, and maintain heightened monitoring.

Five preventive measures: (1) immutable backups with separate credentials and retention enforced at the destination, which is the only real defense against ransomware; (2) a quarterly restore test with the time measured; (3) an alert on write volume outside the hourly pattern, which would have warned at 03:20 instead of 08:15; (4) noexec and ProtectSystem=strict to prevent execution from data paths; and (5) a written and rehearsed response procedure, with the phone numbers, the location of the backups and the steps in order, because at 08:15 in a real incident there is no improvising.

Without logging there is no detection, no investigation, no sensible containment and no learning: the preventive controls of the three previous lessons need detective controls alongside them, because prevention fails at some point and, when it fails silently, the attacker stays for weeks. In Linux, that logging has two faces: syslog, with its facilities and priorities — auth and authpriv are the first ones you look at — and its value as the industry's common language; and journald, binary and indexed, whose decisive advantage is that it adds the metadata itself and the sender cannot lie about it, which makes journalctl _UID=990 --since "-24h" worth more than any grep. The first thing to verify on a server is that the journal is persistent, because in /run it is lost entirely on reboot, and the second is that the real retention covers the investigation window you need — 90 days, not seven.

An application's logging is designed: timestamp with time zone, an event from a closed vocabulary, identity, origin, object, result always, correlation identifier and severity, in structured format. And with an exclusion list as important as the inclusion list: never passwords — failures are usually typos on the right one — never tokens or keys, and personal data minimized to identifiers with a separate translation table; all of it with the retention policy agreed in writing with compliance, because an IP address is personal data. Rotation with logrotate avoids filling the disk, and its two critical directives are create, which preserves permissions and owner, and postrotate with reload, because the process keeps writing to the old inode through its open descriptor. auditd adds what the kernel sees whether the program likes it or not — watching secrets.conf including reads, sudoers, privileged execve, modules and mounts, with -e 2 to make the rules immutable — with the golden rule of auditing the rare and not the frequent, because a rule on the ingestor's writes would generate millions of events a day.

On integrity, the statement to internalize is a hard one: an attacker with root modifies any local log, so a local log is never sufficient proof. chattr +a, the permissions and FSS sealing provide protection against accidents and detection after the fact; the only thing that really protects is immediate shipping to a collector with send-only credentials, because it takes the evidence out of the attacker's reach at the moment it is generated. On the host, AIDE detects changes in binaries and configuration — with rules by file type, sensible exclusions and the reference baseline off the machine — and the manual indicators cover the rest: processes with exe (deleted), unexpected ports, new setuid files and capabilities, unknown jobs and units, and recent SSH keys.

And the response cycle in six phases — preparation, detection and analysis, containment, eradication, recovery, lessons learned — with its two capital mistakes clearly identified: shutting the server down, which destroys the memory, the connections, /tmp and the deleted binaries reachable only through /proc, when isolating the network contains things just as well and preserves everything; and killing the process, when kill -STOP freezes it and leaves it inspectable. The 03:12 case showed the full method: a snapshot of the volatile state first because of the order of volatility, reading /proc/<PID>/ to discover the deleted binary and UID 990, recovering the binary from /proc before the process dies, the time window in journalctl and ausearch, the hunt for persistence against the reference lists, containment without destruction and, at the end, a table of concrete preventive measures with an owner and a date. Evidence preservation — order of volatility, an image with a checksum, always working on the copy, a chain of custody with no gaps — and the legal warning: faced with a breach involving personal data there are short notification deadlines and decisions that are not technical, so legal counsel and the data protection officer come in from the very first minute.

It is worth looking at the whole route, because the module has had a very clear thread: from the mechanisms to the practice, and from prevention to detection.

We started with the framework (05-01): the distinction between protection — an internal, demonstrable mechanism — and security — a global property in the face of an adversary — which imposes the rule that no mechanism is a solution and all of them are layers. We saw the four concepts that describe any access control — subjects, objects, rights and protection domains, where the interesting part is the domain changes — the access matrix with its only two possible implementations — ACLs by columns and capabilities by rows, with the discovery that a file descriptor is a capability — the eight Saltzer and Schroeder principles applied one by one to Meteora, and the four models DAC, MAC, RBAC and ABAC. And the three mechanisms Linux layers on top of the nine bits: capabilities, which slice root's "all or nothing"; SELinux and AppArmor, which add a mandatory check not even root gets around; and seccomp, which reduces the syscall surface from 350 to 40. We closed with the TCB, the attack surface and the confused deputy.

Then we came down to identity (05-02): the UID as the real identity, the three UIDs of every process and the pattern of dropping privilege, the three /etc files field by field, the lifecycle of an account with its two risk points — the accumulation of privileges and the incomplete offboarding, where authorized_keys is what everyone forgets — the KDFs with salt and cost that protect passwords, PAM with its four stacks, SSH with a public key, and sudo with its subtlest lesson: a rule does not constrain what the user can do, but what they can execute.

Then the adversary and hardening (05-03): the catalog of threats with its observable traces and the four common axes — process, port, file, connection — the threat model as the exercise that gives you a criterion for prioritizing, the classes of vulnerability explained by their root cause, the system's defenses (ASLR, NX, canaries, PIE, RELRO) as layers that raise the cost but do not fix anything, and the ordered hardening of meteo-01: updates, minimum surface, a firewall with policy drop, isolation with systemd — the block with the most value per line written — mounting, limits, encryption, secrets and 3-2-1 backups with their restore test.

And this last lesson has answered how you know something has happened, and what to do then.

If Module 4 answered how something is stored so that it is still there tomorrow, Module 5 has answered how you guarantee that only those who should can touch it, and how you know whether anyone tried. The answer has always had the same shape: independent layers, each bounding a different surface, none sufficient on its own, and all of them verifiable with a specific command.

But notice what we have taken for granted for five whole lessons. We have protected meteo-01 as if it were one machine: one kernel, one file system, a set of processes that share everything except what we have separated by hand with permissions, capabilities, profiles and systemd directives. Every layer in this module has been, at bottom, an attempt to simulate isolation inside a system that is not isolated. ProtectSystem=strict pretends the file system is read-only; PrivateTmp pretends /tmp is private; IPAddressDeny pretends the network does not exist.

What if the isolation did not have to be pretended? What if meteo-api could run with its own file system, its own process table, its own network, without even seeing that the rest exists? What if the kernel itself could be duplicated, so that a total compromise of one operating system would not reach the one next door? That is no longer access control: it is isolation, and it is the next level of defense — as well as the foundation on which all modern cloud computing runs.

That is Module 6: Virtualization and Containers, and it begins with Virtualization: Hypervisors and Virtual Machines.