Marta writes at 09:12: "Last night, around three, the website was throwing errors. What happened?". That question either has an exact answer or it has none, and the difference is made by what you did before it happened. A server remembers nothing: the only thing left of the early hours is whatever was written down somewhere, with the right timestamp, without having been rotated, truncated or lost in the reboot. This lesson is about that. And it settles, along the way, the first debt I announced when closing Module 4: /var/log/tramontana/access.log is 412 lines long and growing with nobody rotating it, and the day it fills /var it will take the whole application down with it.

1. The two logging layers that coexist on Ubuntu
2. syslog facilities and severities
3. journalctl in depth
4. Journal persistence
5. /etc/systemd/journald.conf and controlling the size
6. The classic files in /var/log
7. Writing to the log from your scripts with logger
8. logrotate: why and how
9. The problem of the open file after rotation
10. Log centralisation
11. What must NEVER end up in a log
12. Tramontana case: rotation and the answer for Marta

On a freshly installed Ubuntu 24.04 there are two logging systems running at the same time, and understanding the division of labour avoids a lot of confusion:

The real flow: everything a systemd-managed service writes to stdout and stderr is captured by journald, along with what arrives through the /dev/log socket and the kernel's messages. Journald stores it with its metadata and, in addition, forwards it to rsyslog (ForwardToSyslog=yes), which writes it into the same old text files. That is why the same message shows up in journalctl and in /var/log/syslog.

Why keep both? The journal is incomparably better for querying — it filters by unit, by priority, by boot, by field — and the text files are better for what you already know how to do with grep, awk and sed, for third-party tools and for remote sending. On a real server you live with both.

That vocabulary comes from the 1980s and is still alive because the whole industry understands it. Every message carries a facility (where it comes from) and a severity (how much it matters).

Common facilities: auth and authpriv (authentication: sudo, sshd), cron, daemon (services), kern (the kernel), mail, syslog, and local0–local7, reserved for your applications. That last one is the practical key: when backup_tramontana.sh writes to the log, it will do so with a local0 facility and a matching severity, and that way you can filter your own messages without dragging in the system's.

It is the day-to-day tool and it is worth learning properly, because it replaces half a dozen greps.

The time expressions accept yesterday, today, tomorrow, now, -30min, "2 days ago" and absolute dates. It is one of the weighty reasons for using the journal: answering "what happened between 3:00 and 3:30" with text files demands an awk with ranges; here it is two options.

journalctl -b -1 answers "the server rebooted on its own last night, why?", and it requires a persistent journal: exactly what the next section is about.

Each journal entry is not a line: it is a set of fields. Discover them with:

The fields beginning with _ are added by journald, and that is why they are trustworthy: an application cannot forge them. They are used as filters:

Combining filters is the essence of the tool: unit + time window + priority + a reproducible format, in a single line.

This is the detail that surprises everybody the first time: by default, on many installations the journal is volatile. It lives in /run/log/journal, which is a tmpfs in RAM, and it is lost entirely on reboot. If the server went down in the early hours and you rebooted it, you have just destroyed the only evidence of why it went down.

If journalctl --list-boots shows only one line, that is the alarm signal. Enabling persistence is trivial and it is one of the first things to do on a new server:

The directory's permissions (2755 root:systemd-journal) are set by systemd-tmpfiles; that SGID from 05-02 is what lets the systemd-journal group — and adm — read the files created inside it.

A persistent journal with no limits is a full /var waiting its turn. The limits are set here:

That last pair deserves a serious warning: with the default values, a service that emits more than 10,000 messages in 30 seconds will see journald silently discard the rest, leaving only a note of how many it suppressed. Right in the middle of a storm of errors, which is when you need them most. If your application can be noisy during an incident, raise RateLimitBurst or disable it (RateLimitIntervalSec=0), accepting the cost in disk.

The last three are binary: a cat on them spits out rubbish and can mess up your terminal. They are read with their own tools:

A string of lastb entries against root from a foreign IP is exactly the kind of signal you will learn to cut off with fail2ban in 06-03.

Your Module 4 scripts write to /var/log/tramontana/cron-backup.log because you did not know how to do anything else. logger sends the message to the same place as the rest of the system, with its priority and its tag:

Now we modify log() and error() in lib/common.sh so that, as well as their file, they write to the journal:

The -- before the message stops a text starting with a hyphen being interpreted as an option, following the discipline from 04-03. And since the script now runs inside tramontana-backup.service (05-05), everything it writes to stdout also ends up in that unit's journal, with its metadata:

With that, the question "was the backup taken last night?" is a command, not an investigation.

An unrotated log grows until it fills its partition. When /var fills up, the application cannot write, the database cannot commit transactions and the system itself stops recording the disaster. Rotation is not tidiness: it is availability.

logrotate runs daily via logrotate.timer (systemd, which you now know how to read), reads /etc/logrotate.conf and everything in /etc/logrotate.d/, and decides for each file whether it is time to rotate.

Always test before trusting, following the convention of simulating before acting:

This connects with what you saw in 05-04 with lsof +L1. When logrotate does mv access.log access.log.1, the process that had it open carries on writing to the same inode: the new file stays at zero for ever and the old one keeps growing, now invisible. There are two solutions and you have to choose with judgement:

The rule: if the application knows how to reopen its log, use the signal; copytruncate is the last resort for software that does not cooperate.

With one server, journalctl is enough. With two it is not: correlating an error from the load balancer with another from the application by SSHing into each machine is unworkable, and besides, local logs disappear when the machine is compromised or lost, which is precisely when they are needed most. An overview of the options, without setting them up here:

• Remote rsyslog: the classic, lightweight route, *.* @@server:6514 over TCP with TLS. Mature and sufficient for many cases.
• systemd-journal-remote / -upload: it preserves the journal's structure across machines, with journalctl -m to query several.
• Observability stacks such as Loki + Promtail + Grafana, or Elasticsearch + Logstash + Kibana: search, dashboards and alerts, at the price of operating an infrastructure that consumes more resources than the service itself.

One principle does not change: the remote destination must be append-only for whoever sends to it. If an attacker who gets into srv-tramontana can delete the logs on the central server, centralisation contributes nothing forensically.

Compliance warning (GDPR). Logs get copied, sent off the machine, kept for months and read by far more people than production data. They must never contain passwords, tokens, API keys, session cookies, card numbers or personal data of Tramontana's guests (full name, email, telephone, identity document). Logging guest_id=1017 is correct; logging guest=Ana Pérez, [email protected] turns your log file into a file containing personal data, with all the obligations that drags along. The retention policy — how long each type of record is kept — is a legal decision before it is a technical one and must be validated by the data protection or compliance officer, bearing in mind that some records have legal minimum retention periods and others have maximums.

Concrete, cheap measures: review what your application writes at debug level before enabling it in production; check that the URLs you log do not carry tokens in the query string; restrict /var/log/tramontana to the adm group with the ACL from 05-02; and set the retention in logrotate and in journald.conf instead of leaving it to chance.

Every line has a reason: rotate 14 with daily gives two weeks of history, which is what was agreed with Marta; maxsize 100M protects against a spike — an attack or a loop of errors can generate 100 MiB in an afternoon and we do not want to wait until tomorrow —; create 0640 svc-tramontana adm reproduces exactly the permissions the application needs in order to write and the adm group needs in order to read; su root adm is mandatory when the directory does not belong to root, or logrotate refuses to act for safety reasons; sharedscripts avoids reloading the service five times (once per file); and the postrotate uses systemctl reload, which the application translates into reopening its log files, avoiding copytruncate and the loss of lines.

A mandatory verification before calling it good:

That last command is the proof that the postrotate worked: if the application had not reopened the file, a (deleted) with NLINK 0 would show up.

Now the 03:00 question is answered in three commands:

Fifteen errors in the small-hours window, all of them the same: db_timeout with active_connections=200, which is exactly the value of max_connections in /etc/tramontana/app.conf. The report for Marta, with the usual structure — what we know, what it protects and what it does not —: "Between 03:00 and 03:30 there were 15 errors of the same type: the application exhausted its limit of 200 database connections and the requests timed out after 30 s. Nine of them ended up returning a 5xx to users. There was no service outage and no confirmed bookings were lost. The logs now rotate daily and are kept for 14 days, so next time we will have the complete trace. What remains is to find out why the connections run out at that hour, and we will measure that in the next review."

• Assuming the journal is persistent. If journalctl --list-boots shows only one boot, you are losing the logs at every reboot. Create /var/log/journal today.
• Rotating without telling the process. The new file stays empty and the old one keeps growing invisibly. lsof +L1 gives it away; use postrotate with reload or, as a last resort, copytruncate.
• Forgetting su in a logrotate.d whose directory is not root's: logrotate rejects it for safety reasons and silently stops rotating. Check it with logrotate -d.
• rotate 0 or a minimal retention "to save disk": on the day of the incident you will have nothing to look at. Agree the retention with whoever is going to need it.
• Ignoring journald's rate limiting. In a storm of errors you can lose exactly the messages you are after. Review RateLimitBurst if your application is noisy.
• cat on wtmp or btmp. They are binary: they are read with last and lastb. And do not log personal data or secrets: it is a leak in text-file format, replicated in every backup.
• Tip: any script that runs on its own must be able to answer "did it run, and with what result?" with one command. If it cannot, it is missing logging.

1. The small-hours window. Write a single command that shows, from the previous boot, only the messages of priority err or worse from tramontana.service produced between 02:00 and 06:00, in ISO-dated format, and another that counts how many there were of each distinct message.
2. Rotation for a third-party log. A tool writes to /var/log/analytics/events.log as the analytics user, does not know how to reopen its file and generates about 30 MiB a day. Write its logrotate.d file justifying every directive, and explain what you lose with the chosen solution.
3. Access audit. With what you have seen, answer: how many failed login attempts were there yesterday, from which IP, and which users were attempted? Give the commands and say where each piece of data lives.

1.

-o cat leaves only the text of the message, which is what makes it possible to group with sort | uniq -c; with the normal format, the timestamp would make every line unique. It is the same counting pattern from 03-05, now applied to the journal.

2.

daily with rotate 7 gives a week, which is enough for 30 MiB a day; maxsize 50M covers a spike without waiting until tomorrow; compress with delaycompress saves disk without compressing the file that may still be being written; su analytics analytics is essential because the directory is not root's; and copytruncate is the only viable option, since the tool does not know how to reopen its log and there is no signal to send it.

What you lose: the lines written between the copy and the truncation, which is a small but real window, and twice the disk space during that instant. create is not used because with copytruncate the original file is never renamed or recreated.

3.

Failed attempts live in two complementary places: /var/log/btmp (binary, read with lastb, storing the user, terminal, IP and time) and ssh.service's journal, which additionally gives the exact message and allows text filtering. Forty-seven attempts from a single IP in one night is not an absent-minded user: it is a brute-force attack, and blocking it automatically is the subject of 06-03.

srv-tramontana now has a memory. You know that on Ubuntu two layers of logging coexist — journald, binary and structured, and rsyslog, in plain text — and why the journal forwards to syslog instead of replacing it; you handle syslog's eight severities and its facilities, including local0–local7 reserved for your own things; and you really do squeeze journalctl: by unit, live with -f, by time window with --since/--until, by priority with -p, by boot with -b -1, by kernel with -k, by structured fields such as _PID and _COMM, with --grep and with the short-iso, cat and json-pretty formats.

You have made the journal persistent — without that, a reboot destroys the evidence — and you have capped it in journald.conf with SystemMaxUse and MaxRetentionSec, knowing that rate limiting can swallow messages right in the middle of a storm of errors. You recognise the classic files in /var/log and you read wtmp and btmp with last and lastb. Your scripts write to the journal through logger from lib/common.sh, and tramontana-backup.service answers "was the backup taken last night?" all by itself.

And the debt from Module 4 is paid: /etc/logrotate.d/tramontana exists, it has been tested in simulation with logrotate -d, it rotates daily keeping fourteen days, it compresses, it recreates the file with 0640 svc-tramontana adm and it reloads the service in postrotate so that no process carries on writing into an orphaned inode. You know what must never end up in a log and that retention is a legal decision before it is a technical one.

The other half of Marta's question remains. You know what happened at three in the morning — fifteen db_timeouts with the 200 connections exhausted — but not why, nor whether the server was tight on CPU, memory or disk at that moment, nor what to compare today's readings against. In System Monitoring and Performance Tuning you will learn the USE method applied to the four resources, you will really interpret the load average, vmstat, free -h, iostat -xz and sar, you will know when the OOM killer has acted and why, you will establish a baseline for srv-tramontana, and you will apply a numbered diagnostic procedure to Marta's complaint that "the website is slow in the mornings" — where you will discover that the backup and those 200 connections have more to do with each other than they appear to.